Ticket #3741 (closed defect: fixed)

Opened 5 years ago

Last modified 2 years ago

Tracforms can't handle mutated vowel

Reported by: didley@gmx.de Assigned to: hasienda
Priority: high Component: TracFormsPlugin
Severity: major Keywords: unicode umlauts
Cc: Trac Release: 0.11

Description

Whe I using mutated vowels in a Form for example ä in Tracforms the message comes up

'ascii' codec can't encode character u'\xf6' in position 0: ordinal not in range(128)

Other plugins or wiki can handle it. Is it possible to fix it?

Attachments

Change History

10/26/10 22:00:27 changed by hasienda

  • keywords set to umlauts.

#5543 mentioned especially German umlauts and has been closed as a duplicate of this ticket.

12/04/10 01:38:06 changed by hasienda

  • priority changed from normal to high.
  • keywords changed from umlauts to unicode umlauts.
  • severity changed from normal to major.

Confirmed, working towards a solution, since I require it too, and that can't be that hard. It might just be about missing unicode encoding for form content in some places.

12/04/10 05:52:06 changed by rjollos

I'll watch for you patch since this is a frequent problem with plugins and I need to wrap my head around how to fix these issues.

12/04/10 14:27:04 changed by hasienda

  • owner changed from rharkins to hasienda.

Sure, this is one of my short-term tasks. Thanks for taking care.

03/07/11 02:00:21 changed by hasienda

  • status changed from new to assigned.

It took me a lot more time for code studies than expected initially. While the code looks really clean, it's not easy to get all details, since it heavily uses modularization and recursions.

03/07/11 02:00:57 changed by hasienda

(In [9931]) TracFormsPlugin: Allow non-ASCII characters in text input and other fields, refs #3741.

X(HT)ML conform unicode character escaping has been largely inspired by

http://www.xml.com/pub/a/2005/06/15/py-xml.html

03/17/11 06:33:08 changed by hasienda

(In [9967]) TracFormsPlugin: Move XML unicode handling into dedicated script, refs #3741.

Minor code cleanup and preparation for ongoing development started as well.

(follow-up: ↓ 9 ) 03/17/11 14:15:12 changed by didley@gmx.de

Now it works for me. Thanx a lot.

didley

(in reply to: ↑ 8 ) 03/18/11 05:47:33 changed by hasienda

Replying to didley@gmx.de:

Now it works for me. Thanx a lot.

Glad to hear that. But as the values are stored unchanged with the numeric unicode-char-escapes I'm still not totally satisfied by this solution. Hence the still open ticket.

It's just fair to tell you, that the case is not fully resolved from my point of view. Especially regarding #3500 I do suspect, that searching a db full of such escaped strings is not as accessible performance-wise as plain strings. So with the last commit I've already prepared to reverse the escaping process by a profiled algorithm selected from a choice of three. Be prepared to do some db cleanup in case I'll implement this for the storage backend. In fact I've postponed my own production roll-out until I've done the search integration (hopefully by next week).

OTOH it might be possible to escape search strings for the Forms realm as well to obsolete the unescaping process. What would be really needed then, is to provide an abstraction layer, that does handle access to TracForms data in db for all access (i.e. from other plugins too).

03/31/11 22:22:28 changed by hasienda

By now it should be quite safe to follow with at least trunk revision [10005], since I'm using that code in production now.

Since I've switched user input handling to use unicode encoding by default, there are few places to worry about, i.e. non-ASCII variables and usernames, but this is not such a big restriction anymore, if it is a real issue at all. Nevertheless I'm looking forwards to a release candidate for a formal 0.3 release, maybe in April.

Once again, German umlauts work flawlessly, and anyone in need of using similar non-ASCII chars should follow and report his/her experience, please. Thanks for taking care. I'll close this ticket in a while, if there's no more complaint related to the topic.

04/30/11 01:11:41 changed by hasienda

  • status changed from assigned to closed.
  • resolution set to fixed.

(In [10143]) TracFormsPlugin: Releasing version 0.3, pushing development to 0.4, closes #3445, #3550, #3741, #4759 and #8258, refs #3388 and #6993.

This is a major release requiring a Trac environment upgrade.

While the parser logic remains unchanged, there is a lot new supplementary funcionality to make TracForms behave more like the existing Trac core resources (ticket, wiki, attachment, ...).

This version performs a series of non-trivial db schema changes, that especially may leave traces of stale forms (i.e. recorded for wiki pages, that don't exist anymore). So please make sure to read the changelog, BACKUP your environment(s) before installing this version as usual and check the new db tables forms, forms_fields and forms_history after upgrading.


Add/Change #3741 (Tracforms can't handle mutated vowel)




Change Properties
Action