[Patch] UnicodeEncodeError on Japanese tag

[nac]

I got UnicodeEncodeError when I clicked Japanese tag in. (Japanese tag itself can be entered and displayed with tag macro like tag cloud.)

Error is

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

Here is traceback;

File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 435, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 205, in dispatch
  resp = chosen_handler.process_request(req)
File "build/bdist.linux-i586/egg/tractags/web_ui.py", line 99, in process_requestFile "build/bdist.linux-i586/egg/tractags/macros.py", line 80, in expand_macroFile "build/bdist.linux-i586/egg/tractags/macros.py", line 80, in <lambda>

And my trac system is

Trac	0.11.4
Python	2.5.4 (r254:67916, May 24 2009, 15:20:27) [GCC 4.1.2(Gentoo 4.1.2 p1.1)]
setuptools	0.6c9
SQLite	3.6.14
pysqlite	2.3.2
Genshi	0.5.1
mod_python	3.3.1
Pygments	1.0
jQuery	1.2.6

[nac]

Here is a simple patch.

$ diff -c macros.py.org  macros.py
*** macros.py.org	2009-07-29 06:51:30.997066248 +0900
--- macros.py	2009-07-29 07:00:59.698785287 +0900
***************
*** 77,83 ****
  
          ul = builder.ul(class_='taglist')
          for resource, tags in sorted(query_result,
!                                      key=lambda r: str(r[0].id)):
              tags = sorted(tags)
              if tags:
                  rendered_tags = [
--- 77,83 ----
  
          ul = builder.ul(class_='taglist')
          for resource, tags in sorted(query_result,
!                                      key=lambda r: str(r[0].id.encode('utf-8'))):
              tags = sorted(tags)
              if tags:
                  rendered_tags = [

[nac]

Using to_unicode function can resolve this problem and it will be better than .encode method.

from trac.util.text import to_unicode

        for resource, tags in sorted(query_result,
                                     #key=lambda r: str(r[0].id)):
                                     key=lambda r: to_unicode(r[0].id)):

[vchoi@…]

This also happens to portuguese characters.

[Steffen Hoffmann]

This has been fixed in trunk with changeset [3875] more than 2 years ago (see #2493).

Anyway, this doesn't went in the other branches and I feel like to_unicode from Trac would be better coding style anyway, as you see much better what happens from it's name.

So I'll change this in trunk and will ask Michael (otaku42), the current maintainer, to backport it to the other branch(es).

[Steffen Hoffmann]

#6908 has been closed as a duplicate of this ticket.

[Michael Renzmann]

Replying to hasienda:

So I'll change this in trunk and will ask Michael (otaku42), the current maintainer, to backport it to the other branch(es).

Yes, please do so. Not sure if I backport that to earlier versions, though. I think it would be good to get a new version out that contains either just recent bug fixes, or maybe also new features such as i18n.

[vchoi@…]

Replying to otaku42:

Yes, please do so. Not sure if I backport that to earlier versions, though. I think it would be good to get a new version out that contains either just recent bug fixes, or maybe also new features such as i18n.

I also think so.

[Steffen Hoffmann]

Replying to otaku42:

Replying to hasienda:

So I'll change this in trunk and will ask Michael (otaku42), the current maintainer, to backport it to the other branch(es).

Yes, please do so. Not sure if I backport that to earlier versions, though. I think it would be good to get a new version out that contains either just recent bug fixes, or maybe also new features such as i18n.

Ok, at least i18n has been added to trunk today, finally, and I'll rework code related to this issue now.

How about doing a 0.6.1 with recent bug fixes and a reserve i18n for the next major release? At least this has been prepared by pushing trunk version to 0.7dev in changeset [9316].

[Steffen Hoffmann]

(In [9322]) TagsPlugin: Rework string encoding for (non-ASCII) tag names, closes #5575.

This has been fixed in other branches in a different way before, however applying a method from Trac could be preferred.

[Michael Renzmann]

Replying to hasienda:

How about doing a 0.6.1 with recent bug fixes and a reserve i18n for the next major release?

No objections from my side on this one.