Modify

Opened 4 years ago

Last modified 4 years ago

#7698 assigned defect

Handle unicode arguments for embedded custom TracQuery too

Reported by: bog@… Owned by: hasienda
Priority: high Component: WikiTicketCalendarMacro
Severity: major Keywords: unicode 0.11 0.12
Cc: rjollos, boris.savelev@… Trac Release: 0.12

Description

I had tried 0.11 (as single file) and 0.12 (as egg install in .../plugins) and 0.12 (as easy_install). In any case i have:

Genshi UnicodeDecodeError error while rendering template (unknown template location)

CentOS 5.4 + http://packages.sw.be/trac/
Apache 2.2.3
trac 0.12
python-setuptools-0.6c5-2.el5
python-genshi-0.6-1.el5.rf
python-sqlite-1.1.7-1.2.1
babel-0.9.5-1.el5
python-babel-0.9.5-1.el5
python-pygments-1.1.1-1.el5

Attachments (1)

20101206_wikiticketcalendar_ru.png (24.3 KB) - added by hasienda 4 years ago.
WikiTicketCalenderMacro v1.2.4dev with locale set to 'ru_RU.UTF-8'

Download all attachments as: .zip

Change History (20)

comment:1 Changed 4 years ago by bog@…

Trac[SoC:formatter] ERROR: Macro WikiTicketCalendar(None) failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 710, in _macro_formatter
    return macro.process(args, in_paragraph=True)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 334, in process
    text = "</p>%s<p>" % _markup_to_unicode(text)
  File "/usr/lib/python2.4/site-packages/trac/wiki/formatter.py", line 82, in _markup_to_unicode
    markup = stream.render('xhtml', encoding=None, strip_whitespace=False)
  File "/usr/lib/python2.4/site-packages/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "/usr/lib/python2.4/site-packages/genshi/output.py", line 57, in encode
    return _encode(''.join(list(iterator)))
  File "/usr/lib/python2.4/site-packages/genshi/output.py", line 371, in __call__
    yield _emit(kind, data, escape(data, quotes=False))
  File "/usr/lib/python2.4/site-packages/genshi/core.py", line 517, in escape
    return cls(text)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

comment:2 Changed 4 years ago by bog@…

httpd.conf

PythonOption TracLocale ru_RU.UTF8

comment:3 Changed 4 years ago by hasienda

  • Cc rjollos added
  • Keywords unicode 0.11 added
  • Summary changed from Genshi UnicodeDecodeError error while rendering template (unknown template location) to Genshi UnicodeDecodeError while rendering

Looks like you've found another unicode-unsafe part, while I meant to have fixed all possibly variables that could need utf8 encoding lately.

There is certainly no template in WikiTicketCalendarMacro right now nor did it exist before, so the Genshi error wording is certainly misleading, if not plain wrong. But we'll see.

Well, would you verify and tell me the exact code version/revision you're applying, please?

comment:4 Changed 4 years ago by bog@…

It works when i comment TracLocale.

Trac  	0.12
Babel 	0.9.5
Genshi 	0.6
mod_python 	3.2.8
Pygments 	1.1.1
pysqlite 	2.3.3
Python 	2.4.3 (#1, Sep 3 2009, 15:37:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)]
pytz 	2010h
setuptools 	0.6c5
SQLite 	3.3.6
Subversion 	1.6.12 (r955767)
jQuery:	1.4.2

wikiticketcalendarmacro-r8937.zip

comment:5 Changed 4 years ago by hasienda

Revision indicates latest version, fine. Hm, problem comes with that locale. Tested locales (see screenshots on wiki page) i.e. didn't change the weekday column headers to something with non-ASCII chars. This might be the problem, but I'll have to test with your locale to be sure. Didn't try before, since currently there are no Russian translations for this plugin.

comment:6 Changed 4 years ago by hasienda

  • Priority changed from normal to high

Make this top priority to investigate and fix, as it prevents use in localized environments, and I really care for that in my own applications as well.

comment:7 follow-up: Changed 4 years ago by anonymous

  • Cc boris.savelev@… added
  • Keywords 0.12 added

I create ru translation, but this doesn't help

I have trac 0.12 with lighttpd installation. I have "bin-environment" => ("LC_TIME" => ru_RU.utf8")
When I comment this setting all works fine.
Problem is that in ru_RU weekdays is in non-ASCII

boris@bsavelev:~$ LC_TIME=ru date
Mon Dec  6 17:47:30 MSK 2010
boris@bsavelev:~$ LC_TIME=ru_RU.UTF-8 date
Пнд Дек  6 17:47:34 MSK 2010

mounth also is not-ASCII

comment:8 Changed 4 years ago by Boris Savelev

here the patch

Index: wikiticketcalendar/macro.py
===================================================================
--- wikiticketcalendar/macro.py (revision 9588)
+++ wikiticketcalendar/macro.py (working copy)
@@ -404,11 +404,11 @@
         heading(align='center')
 
         for day in calendar.weekheader(2).split()[:-2]:
-            col = tag.th(day)
+            col = tag.th(to_unicode(day))
             col(class_='workday', scope='col')
             heading(col)
         for day in calendar.weekheader(2).split()[-2:]:
-            col = tag.th(day)
+            col = tag.th(to_unicode(day))
             col(class_='weekend', scope='col')
             heading(col)

comment:9 Changed 4 years ago by Boris Savelev

#: wikiticketcalendar/macro.py:435                                                                                                          
#, python-format                                                                                                                            
msgid "Go to page %s"                                                                                                                       
msgstr "Переход на страницу %s"                                                                                                             
                                                                                                                                            
#: wikiticketcalendar/macro.py:442                                                                                                          
#, python-format                                                                                                                            
msgid "Create page %s"                                                                                                                      
msgstr "Создать страницу %s"

also apply ru.po please

comment:10 follow-up: Changed 4 years ago by anonymous

another unicode error:
self.config.get('wikiticketcalendar'... return unicode object
format_date expect str object

patch:

@@ -497,7 +497,7 @@
                                     continue
                             else:
                                 duedate = format_date(day_dt,
-                                                      self.due_field_fmt)
+                                                      str(self.due_field_fmt))
                                 if not due == duedate:
                                     continue

comment:11 in reply to: ↑ 7 Changed 4 years ago by hasienda

  • Status changed from new to assigned

Replying to anonymous:

I create ru translation, but this doesn't help

This is what I've been waiting for, thanks. I does help, since now there is something for testing with non-ASCII days and months name.

comment:12 in reply to: ↑ 10 Changed 4 years ago by hasienda

Replying to anonymous:

another unicode error:
self.config.get('wikiticketcalendar'... return unicode object
format_date expect str object

Oh, never expected that, really curious to see, what's your value/setting there. Would you mind to share it?

comment:13 Changed 4 years ago by bsavelev

  • Resolution set to fixed
  • Status changed from assigned to closed
Error: Macro WikiTicketCalendar(cdate=1, query=type=суд) failed

strftime() argument 1 must be str, not unicode

I receive this error then set

[wikiticketcalendar]
ticket.due_field.name = next_session
ticket.due_field.format = %d.%m.%y

then comment ticket.due_field.format all works fine

env

Trac 	0.12
Babel 	0.9.5
Docutils 	0.5
Genshi 	0.6
MySQL 	server: "5.0.51a-24+lenny4", client: "5.0.51a", thread-safe: 1
MySQLdb 	1.2.2
Pygments 	0.10
Python 	2.5.2 (r252:60911, Jan 24 2010, 17:44:40) [GCC 4.3.2]
pytz 	2008c
setuptools 	0.6c8
Subversion 	1.5.1 (r32289)
jQuery:	1.3.2

trac with fastcgi on lighttpd

env for fastcgi

TRAC_ENV=...
PYTHON_EGG_CACHE=...
LC_TIME=ru_RU.UTF-8

PS:
I novice in python, are my patches correct?

comment:14 Changed 4 years ago by bsavelev

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:15 Changed 4 years ago by bsavelev

  • Status changed from reopened to new

comment:16 Changed 4 years ago by hasienda

(In [9596]) WikiTicketCalendarMacro: Add more changes to handle unicode strings properly, refs #7698.

Further improvements to handle issues with non-ASCII day and month names,
that appear i.e. when locale has been set to ru_RU.UTF8,
and Russian translation came from Boris Savelev too.
Bumping version on trunk branch.

Changed 4 years ago by hasienda

WikiTicketCalenderMacro v1.2.4dev with locale set to 'ru_RU.UTF-8'

comment:17 Changed 4 years ago by hasienda

  • Status changed from new to assigned

WikiTicketCalenderMacro v1.2.4dev with locale set to 'ru_RU.UTF-8'

Nice, thanks again to Boris for taking the time to investigate this.

Beware: When testing, I recognized, that you need the locale in question as a pre-requisite to see anything. In fact all Trac pages disappeared after setting 'ru_RU.UTF-8' and before installing that locale at my test system.

But I just see, we're not finished yet, since there's come more ...

comment:18 Changed 4 years ago by bsavelev

great! thanks a lot!

comment:19 Changed 4 years ago by hasienda

  • Summary changed from Genshi UnicodeDecodeError while rendering to Handle unicode arguments for embedded custom TracQuery too

I nearly forgot, what was the remaining issue, so this is more obvious now.

Add Comment

Modify Ticket

Action
as assigned .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.