UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 4: ordinal not in range(128)

[anonymous]

when clicking on the menu item "hours" the error

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 4: ordinal not in range(128)

[Jeff Hammel]

not enough information to reproduce; please reopen with a stack trace, etc, and i'll look at it. maybe this is related to #4377 ? or maybe not. impossible to tell with so little info. suffice it to say, i can't reproduce and clicking on the hours menu item worksforme

[anonymous]

see here: ​t:#7959

[anonymous]

After adding #7959, the error changed to:

Genshi UnicodeDecodeError error while rendering template 'C:\Python25\lib\site-packages\trachoursplugin-0.3-py2.5.egg\trachours\templates\hours_timeline.html', line 96, char -1

I'm using a Japanese version of Windows and was wondering if that has anything to do with it.

[anonymous]

The dump log for the above error is as follows:

2009-03-02 17:34:56,733 Trac[main] DEBUG: Dispatching <Request "GET u'/hours'"> 2009-03-02 17:34:56,750 Trac[hours] DEBUG: QueryModule: Using default query: status!=bogus 2009-03-02 17:34:56,750 Trac[chrome] DEBUG: Prepare chrome data for request 2009-03-02 17:34:56,765 Trac[session] DEBUG: Retrieving session for ID u'Administrator' 2009-03-02 17:34:56,765 Trac[query] DEBUG: Count results in Query: 6 2009-03-02 17:34:56,765 Trac[query] DEBUG: Query SQL: SELECT t.id AS id,t.summary AS summary,t.status AS status,t.priority AS priority,t.time AS time,t.changetime AS changetime,priority.value AS priority_value,estimatedhours.value AS estimatedhours FROM ticket AS t

LEFT OUTER JOIN ticket_custom AS estimatedhours ON (id=estimatedhours.ticket AND estimatedhours.name='estimatedhours') LEFT OUTER JOIN enum AS priority ON (priority.type='priority' AND priority.name=priority)

WHERE COALESCE(t.status,)!=u'bogus' ORDER BY COALESCE(priority.value,)=,CAST(priority.value AS int),t.id 2009-03-02 17:34:57,108 Trac[main] WARNING: 500 Trac Error (Genshi UnicodeDecodeError error while rendering template 'C:\Python25\lib\site-packages\trachoursplugin-0.3-py2.5.egg\trachours\templates\hours_timeline.html', line 96, char -1)

[public@…]

I got this error, too.

I've just installed trac-0.11.3 and the trachoursplugin downloaded today from scratch and went to the "/hours" page and got exactly the same error:

Genshi UnicodeDecodeError error while rendering template '[...]/trachours/templates/hours_timeline.html', line 96, char -1

Lines 95-96 of the template read

	<a href="${prev_url}">Week of ${prev_week.strftime('%B %d, %Y')}
	  - ${from_date.strftime('%B %d, %Y')}</a>

Apparently, strftime returns non-ASCII characters. On my system with German locale, the month name (%B) of May is "März":

$ python
Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> import os
>>> import time
>>> print os.environ['LANG']
de_DE.UTF-8
>>> locale.setlocale(locale.LC_ALL, 'de_DE.UTF-8')
'de_DE.UTF-8'
>>> time.strftime('%B %d, %Y')
'M\xc3\xa4rz 18, 2009'
>>> locale.setlocale(locale.LC_ALL, 'en_US.UTF-8')
'en_US.UTF-8'
>>> time.strftime('%B %d, %Y')
'March 18, 2009'
>>> 

Here you can see that non-ASCII characters were indeed returned (as a non-unicode string in this case). Setting LANG=en_US.UTF-8 works around the bug on my system. I don't know what the proper fix for this is, so I'm leaving this to you.

By the way, the setlocale() call is probably located in trac/web/main.py, dispatch_request():

def dispatch_request(environ, start_response):
    ...
    locale.setlocale(locale.LC_ALL, environ['trac.locale'])

Also, you can get the locale's encoding (which should be the encoding used by strftime()) with locale.getlocale():

$ python
Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'de_DE.UTF-8')
'de_DE.UTF-8'
>>> locale.getlocale()
('de_DE', 'UTF8')
>>> 

I hope that's enough info to make fixing the bug easy.

Regards, Milan

[Jeff Hammel]

Thanks for the helpful information. I think I understand the issue for the first time (outside of not understanding the strangeness that is unicode in python, but that's another story). I may have a solution: the Trac to_unicode helper function. This is actually ultimately a python "bug", or at least issue, that is more thoroughly documented here:

​http://bugs.python.org/issue5398

Will commit shortly. I have no good way of testing this, unless someone wants to hold my hand through the process, so if this does not work, I encourage patches (hopefully, patches that will work in the generic sense and not for particular use-cases) as I don't have a setup to debug.

[Jeff Hammel]

(In [5374]) use to_unicode helper function; hopefully fixes #4420