Opened 8 years ago

Closed 6 years ago

#3741 closed defect (fixed)

Tracforms can't handle mutated vowel

Reported by: didley@… Owned by: Steffen Hoffmann
Priority: high Component: TracFormsPlugin
Severity: major Keywords: unicode umlauts
Cc: Trac Release: 0.11


Whe I using mutated vowels in a Form for example ä in Tracforms the message comes up

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

Other plugins or wiki can handle it. Is it possible to fix it?

Attachments (0)

Change History (11)

comment:1 Changed 6 years ago by Steffen Hoffmann

Keywords: umlauts added

#5543 mentioned especially German umlauts and has been closed as a duplicate of this ticket.

comment:2 Changed 6 years ago by Steffen Hoffmann

Keywords: unicode added
Priority: normalhigh
Severity: normalmajor

Confirmed, working towards a solution, since I require it too, and that can't be that hard. It might just be about missing unicode encoding for form content in some places.

comment:3 Changed 6 years ago by Ryan J Ollos

I'll watch for you patch since this is a frequent problem with plugins and I need to wrap my head around how to fix these issues.

comment:4 Changed 6 years ago by Steffen Hoffmann

Owner: changed from Rich Harkins to Steffen Hoffmann

Sure, this is one of my short-term tasks. Thanks for taking care.

comment:5 Changed 6 years ago by Steffen Hoffmann

Status: newassigned

It took me a lot more time for code studies than expected initially. While the code looks really clean, it's not easy to get all details, since it heavily uses modularization and recursions.

comment:6 Changed 6 years ago by Steffen Hoffmann

(In [9931]) TracFormsPlugin: Allow non-ASCII characters in text input and other fields, refs #3741.

X(HT)ML conform unicode character escaping has been largely inspired by

comment:7 Changed 6 years ago by Steffen Hoffmann

(In [9967]) TracFormsPlugin: Move XML unicode handling into dedicated script, refs #3741.

Minor code cleanup and preparation for ongoing development started as well.

comment:8 Changed 6 years ago by didley@…

Now it works for me. Thanx a lot.


comment:9 in reply to:  8 Changed 6 years ago by Steffen Hoffmann

Replying to

Now it works for me. Thanx a lot.

Glad to hear that. But as the values are stored unchanged with the numeric unicode-char-escapes I'm still not totally satisfied by this solution. Hence the still open ticket.

It's just fair to tell you, that the case is not fully resolved from my point of view. Especially regarding #3500 I do suspect, that searching a db full of such escaped strings is not as accessible performance-wise as plain strings. So with the last commit I've already prepared to reverse the escaping process by a profiled algorithm selected from a choice of three. Be prepared to do some db cleanup in case I'll implement this for the storage backend. In fact I've postponed my own production roll-out until I've done the search integration (hopefully by next week).

OTOH it might be possible to escape search strings for the Forms realm as well to obsolete the unescaping process. What would be really needed then, is to provide an abstraction layer, that does handle access to TracForms data in db for all access (i.e. from other plugins too).

comment:10 Changed 6 years ago by Steffen Hoffmann

By now it should be quite safe to follow with at least trunk revision [10005], since I'm using that code in production now.

Since I've switched user input handling to use unicode encoding by default, there are few places to worry about, i.e. non-ASCII variables and usernames, but this is not such a big restriction anymore, if it is a real issue at all. Nevertheless I'm looking forwards to a release candidate for a formal 0.3 release, maybe in April.

Once again, German umlauts work flawlessly, and anyone in need of using similar non-ASCII chars should follow and report his/her experience, please. Thanks for taking care. I'll close this ticket in a while, if there's no more complaint related to the topic.

comment:11 Changed 6 years ago by Steffen Hoffmann

Resolution: fixed
Status: assignedclosed

(In [10143]) TracFormsPlugin: Releasing version 0.3, pushing development to 0.4, closes #3445, #3550, #3741, #4759 and #8258, refs #3388 and #6993.

This is a major release requiring a Trac environment upgrade.

While the parser logic remains unchanged, there is a lot new supplementary funcionality to make TracForms behave more like the existing Trac core resources (ticket, wiki, attachment, ...).

This version performs a series of non-trivial db schema changes, that especially may leave traces of stale forms (i.e. recorded for wiki pages, that don't exist anymore). So please make sure to read the changelog, BACKUP your environment(s) before installing this version as usual and check the new db tables forms, forms_fields and forms_history after upgrading.

Modify Ticket

as closed The owner will remain Steffen Hoffmann.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment

E-mail address and name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.