Tracforms can't handle mutated vowel

[didley@…]

Whe I using mutated vowels in a Form for example ä in Tracforms the message comes up

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

Other plugins or wiki can handle it. Is it possible to fix it?

[Steffen Hoffmann]

#5543 mentioned especially German umlauts and has been closed as a duplicate of this ticket.

[Steffen Hoffmann]

Confirmed, working towards a solution, since I require it too, and that can't be that hard. It might just be about missing unicode encoding for form content in some places.

[Ryan J Ollos]

I'll watch for you patch since this is a frequent problem with plugins and I need to wrap my head around how to fix these issues.

[Steffen Hoffmann]

Sure, this is one of my short-term tasks. Thanks for taking care.

[Steffen Hoffmann]

It took me a lot more time for code studies than expected initially. While the code looks really clean, it's not easy to get all details, since it heavily uses modularization and recursions.

[Steffen Hoffmann]

(In [9931]) TracFormsPlugin: Allow non-ASCII characters in text input and other fields, refs #3741.

X(HT)ML conform unicode character escaping has been largely inspired by

​http://www.xml.com/pub/a/2005/06/15/py-xml.html

[Steffen Hoffmann]

(In [9967]) TracFormsPlugin: Move XML unicode handling into dedicated script, refs #3741.

Minor code cleanup and preparation for ongoing development started as well.

[didley@…]

Now it works for me. Thanx a lot.

didley

[Steffen Hoffmann]

Replying to didley@gmx.de:

Now it works for me. Thanx a lot.

Glad to hear that. But as the values are stored unchanged with the numeric unicode-char-escapes I'm still not totally satisfied by this solution. Hence the still open ticket.

It's just fair to tell you, that the case is not fully resolved from my point of view. Especially regarding #3500 I do suspect, that searching a db full of such escaped strings is not as accessible performance-wise as plain strings. So with the last commit I've already prepared to reverse the escaping process by a profiled algorithm selected from a choice of three. Be prepared to do some db cleanup in case I'll implement this for the storage backend. In fact I've postponed my own production roll-out until I've done the search integration (hopefully by next week).

OTOH it might be possible to escape search strings for the Forms realm as well to obsolete the unescaping process. What would be really needed then, is to provide an abstraction layer, that does handle access to TracForms data in db for all access (i.e. from other plugins too).

[Steffen Hoffmann]

By now it should be quite safe to follow with at least trunk revision [10005], since I'm using that code in production now.

Since I've switched user input handling to use unicode encoding by default, there are few places to worry about, i.e. non-ASCII variables and usernames, but this is not such a big restriction anymore, if it is a real issue at all. Nevertheless I'm looking forwards to a release candidate for a formal 0.3 release, maybe in April.

Once again, German umlauts work flawlessly, and anyone in need of using similar non-ASCII chars should follow and report his/her experience, please. Thanks for taking care. I'll close this ticket in a while, if there's no more complaint related to the topic.

[Steffen Hoffmann]

(In [10143]) TracFormsPlugin: Releasing version 0.3, pushing development to 0.4, closes #3445, #3550, #3741, #4759 and #8258, refs #3388 and #6993.

This is a major release requiring a Trac environment upgrade.

While the parser logic remains unchanged, there is a lot new supplementary funcionality to make TracForms behave more like the existing Trac core resources (ticket, wiki, attachment, ...).

This version performs a series of non-trivial db schema changes, that especially may leave traces of stale forms (i.e. recorded for wiki pages, that don't exist anymore). So please make sure to read the changelog, BACKUP your environment(s) before installing this version as usual and check the new db tables forms, forms_fields and forms_history after upgrading.