Opened 8 years ago

Closed 5 years ago

Tracforms can't handle mutated vowel

Reported by: Owned by: didley@… hasienda high TracFormsPlugin major unicode umlauts 0.11

Description

Whe I using mutated vowels in a Form for example ä in Tracforms the message comes up

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)


Other plugins or wiki can handle it. Is it possible to fix it?

comment:1 Changed 6 years ago by hasienda

#5543 mentioned especially German umlauts and has been closed as a duplicate of this ticket.

comment:2 Changed 5 years ago by hasienda

• Priority changed from normal to high
• Severity changed from normal to major

Confirmed, working towards a solution, since I require it too, and that can't be that hard. It might just be about missing unicode encoding for form content in some places.

comment:3 Changed 5 years ago by rjollos

I'll watch for you patch since this is a frequent problem with plugins and I need to wrap my head around how to fix these issues.

comment:4 Changed 5 years ago by hasienda

• Owner changed from rharkins to hasienda

Sure, this is one of my short-term tasks. Thanks for taking care.

comment:5 Changed 5 years ago by hasienda

• Status changed from new to assigned

It took me a lot more time for code studies than expected initially. While the code looks really clean, it's not easy to get all details, since it heavily uses modularization and recursions.

comment:6 Changed 5 years ago by hasienda

(In [9931]) TracFormsPlugin: Allow non-ASCII characters in text input and other fields, refs #3741.

X(HT)ML conform unicode character escaping has been largely inspired by

comment:7 Changed 5 years ago by hasienda

(In [9967]) TracFormsPlugin: Move XML unicode handling into dedicated script, refs #3741.

Minor code cleanup and preparation for ongoing development started as well.

comment:8 follow-up: ↓ 9 Changed 5 years ago by didley@…

Now it works for me. Thanx a lot.

didley

comment:9 in reply to: ↑ 8 Changed 5 years ago by hasienda

Now it works for me. Thanx a lot.

Glad to hear that. But as the values are stored unchanged with the numeric unicode-char-escapes I'm still not totally satisfied by this solution. Hence the still open ticket.

It's just fair to tell you, that the case is not fully resolved from my point of view. Especially regarding #3500 I do suspect, that searching a db full of such escaped strings is not as accessible performance-wise as plain strings. So with the last commit I've already prepared to reverse the escaping process by a profiled algorithm selected from a choice of three. Be prepared to do some db cleanup in case I'll implement this for the storage backend. In fact I've postponed my own production roll-out until I've done the search integration (hopefully by next week).

OTOH it might be possible to escape search strings for the Forms realm as well to obsolete the unescaping process. What would be really needed then, is to provide an abstraction layer, that does handle access to TracForms data in db for all access (i.e. from other plugins too).

comment:10 Changed 5 years ago by hasienda

By now it should be quite safe to follow with at least trunk revision [10005], since I'm using that code in production now.

Since I've switched user input handling to use unicode encoding by default, there are few places to worry about, i.e. non-ASCII variables and usernames, but this is not such a big restriction anymore, if it is a real issue at all. Nevertheless I'm looking forwards to a release candidate for a formal 0.3 release, maybe in April.

Once again, German umlauts work flawlessly, and anyone in need of using similar non-ASCII chars should follow and report his/her experience, please. Thanks for taking care. I'll close this ticket in a while, if there's no more complaint related to the topic.

comment:11 Changed 5 years ago by hasienda

• Resolution set to fixed
• Status changed from assigned to closed

(In [10143]) TracFormsPlugin: Releasing version 0.3, pushing development to 0.4, closes #3445, #3550, #3741, #4759 and #8258, refs #3388 and #6993.

This is a major release requiring a Trac environment upgrade.

While the parser logic remains unchanged, there is a lot new supplementary funcionality to make TracForms behave more like the existing Trac core resources (ticket, wiki, attachment, ...).

This version performs a series of non-trivial db schema changes, that especially may leave traces of stale forms (i.e. recorded for wiki pages, that don't exist anymore). So please make sure to read the changelog, BACKUP your environment(s) before installing this version as usual and check the new db tables forms, forms_fields and forms_history after upgrading.