UnicodeError: source returned bytes, but no encoding specified

[porqpynz@…]

How to Reproduce

While doing a GET operation on /roadmap, Trac issued an internal error.

User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/601.2.7 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.7

System Information

Trac	1.0.2
Babel	1.3
CustomFieldAdmin	0.2.6
Docutils	0.12
Genshi	0.7 (with speedups)
GIT	2.1.4
mod_wsgi	4.3.0 (WSGIProcessGroup WSGIApplicationGroup %{GLOBAL})
Pygments	2.0.1
pysqlite	2.6.0
Python	2.7.9 (default, Mar 1 2015, 13:01:26) [GCC 4.9.2]
pytz	2012c
setuptools	5.5.1
SQLite	3.8.7.1
Subversion	1.8.10 (r1615264)
jQuery	1.7.2
jQuery UI	1.10.1
jQuery Timepicker	1.2

Enabled Plugins

NavAdd	0.3
SimpleMultiProject	0.5.1dev
Trac-jsGantt	0.9
TracCustomFieldAdmin	0.2.6
TracMasterTickets	3.0.2
TracSubTicketsPlugin	0.1.1.dev-20120115

Python Traceback

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/trac/web/main.py", line 512, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.7/dist-packages/trac/web/main.py", line 240, in dispatch
    content_type)
  File "/usr/lib/python2.7/dist-packages/trac/web/chrome.py", line 1009, in render_template
    stream |= self._filter_stream(req, method, filename, stream, data)
  File "/usr/lib/python2.7/dist-packages/genshi/core.py", line 133, in __or__
    return Stream(_ensure(function(self)), serializer=self.serializer)
  File "/usr/lib/python2.7/dist-packages/trac/web/chrome.py", line 1192, in inner
    data)
  File "build/bdist.linux-x86_64/egg/simplemultiproject/roadmap.py", line 113, in filter_stream
    stream = stream | filter_.replace(HTML('<label for="showcompleted">Show completed milestones and '
  File "/usr/lib/python2.7/dist-packages/genshi/input.py", line 442, in HTML
    return Stream(list(HTMLParser(BytesIO(text), encoding=encoding)))
  File "/usr/lib/python2.7/dist-packages/genshi/core.py", line 273, in _ensure
    event = stream.next()
  File "/usr/lib/python2.7/dist-packages/genshi/input.py", line 449, in _coalesce
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/python2.7/dist-packages/genshi/input.py", line 337, in _generate
    raise UnicodeError("source returned bytes, but no encoding specified")
UnicodeError: source returned bytes, but no encoding specified

[Ryan J Ollos]

Also reported in ​gdiscussion:trac-users:GOSrrAJPSOk.

[falkb]

Just a guess without testing the code change here: Could you try something like this in line 113 of roadmap.py, please?:

+ stream = HTML(to_unicode(stream))
  stream = stream | filter_.replace(HTML('<label for="showcompleted">Show completed milestones and '
                                         'versions</label>'))

The old code before the latest changes still had that additional function call to_unicode() wrapped by function HTML().

[porqpynz@…]

I'll try that today.

[porqpynz@…]

Tried the code above. I added stream = HTML(to_unicode(stream)) as line 113 in roadmap.py. No joy. I'm getting the same error. Perhaps I did this wrong! (I'm not a Python programmer :).

Python Traceback
Most recent call last:

    File "/usr/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/main.py", line 497, in _dispatch_request
    File "/usr/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/main.py", line 233, in dispatch
    File "/usr/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/chrome.py", line 983, in render_template
    File "/usr/lib64/python2.6/site-packages/Genshi-0.7-py2.6-linux-x86_64.egg/genshi/core.py", line 133, in __or__
    File "/usr/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/chrome.py", line 1162, in inner
    File "build/bdist.linux-x86_64/egg/simplemultiproject/roadmap.py", line 114, in filter_stream

    Code fragment:
    Line	
    109	 
    110	        if filename == 'roadmap.html':
    111	            # Change label to include versions
    112	            filter_ = Transformer('//label[@for="showcompleted"]')
    113	            # test bug fix in line below - -=WFA=-
    114	            stream = HTML(to_unicode(stream))
    115	            stream = stream | filter_.replace(HTML('<label for="showcompleted">Show completed milestones and '
    116	                                                   'versions</label>'))
    117	            # Add additional checkboxes to preferences
    118	            data['smp_render'] = 'prefs'  # specify which part of template to render
    119	 

    Local variables:
    Name	Value
    data 	{'abs_url_of': <function get_abs_url at 0x7f8fb2d52e60>, 'all': <built-in ...
    filename 	'roadmap.html'
    filter_ 	<genshi.filters.transform.Transformer object at 0x7f8fb2d93360>
    method 	'xhtml'
    req 	<RequestWithSession "GET '/roadmap'">
    self 	<simplemultiproject.roadmap.SmpRoadmapGroup object at 0x7f8fb2814f90>
    stream 	<genshi.core.Stream object at 0x7f8fb2a29050>

[porqpynz@…]

I forgot to add the most important part of the error message!

Trac detected an internal error:

NameError: global name 'to_unicode' is not defined

[porqpynz@…]

I guess I'll leave this to more knowledgeable programmers like the maintainer(s). :)

[Cinc-th]

In the meantime just disable the SmpRoadmapGroup component so you get access to the roadmap again. This is more a nice to have component but not essential.

[Cinc-th]

Oh, and you may use version 0.0.4 in the tags subdirectory. Despite the huge version jump it has all the features of the current trunk.

Regards

[Cinc-th]

V0.0.4 in tags won't help you...

I committed [15049] which may fix your problem with the current trunk. At least it works over here with Trac 1.0.9 and Genshi 0.7. Before I could reproduce your problem.

[Ryan J Ollos]

Looks like that is the right fix. See also #10295 and ​wiki:TracDev/ApiChanges/1.0#Genshimandatory.

[Cinc-th]

Replying to rjollos:

Looks like that is the right fix. See also #10295 and ​wiki:TracDev/ApiChanges/1.0#Genshimandatory.

Ah, there it is :-).

I knew I had seen some info about Genshi 0.7 but couldn't find it again... Thanks for the pointer.

[falkb]

Please, give me a bit survey: Does it work with Genshi 0.6 again? Which branches are affected? Which branches have a bugfix commit now? (trunk, smp-0.0.4, smp-0.5.0)

[Cinc-th]

Should still work with Genshi 0.6 (not tested yet).

Only trunk got the fix.

[falkb]

Replying to Cinc-th:

Should still work with Genshi 0.6 (not tested yet).

Only trunk got the fix.

User oneman reported in the "trac-users"-mailinglist that error with a plugin version 0.5.1dev, what version is that? In case, it's that "smp-0.5.0"-tag, we should fix it also there.

[Cinc-th]

That's the current trunk. Never, ever change a tag version. For changes in an older version we had to create a branch.

[falkb]

Replying to Cinc-th:

That's the current trunk. Never, ever change a tag version. For changes in an older version we had to create a branch.

Just in the rare case, a tag does not work at all, it should be moved to a new (working) source state. This was my background idea here.

Uhmm, if user oneman tried the trunk then it's his own bad luck in a certain sense, because it's a developer playground. :-P

[Cinc-th]

Uhmm, if user oneman tried the trunk then it's his own bad luck in a certain sense, because it's a developer playground. :-P

Yeah, but in this case the problem was introduced weeks before so the tagged versions won't help either. Anyway, I've got 1.0.9 up and running as a test system now and will test a little bit more.

[Cinc-th]

[15049] seems to fix the problem. Solved the problem for the user from the thread mentioned in comment:3.