Ordinal out of range when including UTF-8 source

[olaf.meeuwissen@…]

I'm trying to include a file from our SVN repository that contains UTF-8 Japanese. The svn:mime-type is set to text/x-trac-wiki. Here is what I (approximately) get:

Error: Macro Include(source:path/to/japanese.txt) failed

ascii<div class="code"><pre>{{{ #!comment 日本語のコメントです。 }}} これはうまくいかないね。
</pre></div>3756ordinal not in range(128)

The number before the ordinal not in range error message depends on the file's content.

Pasting the file's content into the wiki page works fine. Including wiki pages with Japanese text works fine too. I had expected the include macro to work fine as well.

[Ryan J Ollos]

I tried to reproduce, but I'm unable to. Are you running 3.0dev-r12030 of the plugin? Are you running Trac 1.0 or 1.0.1?

Would it be possible for you to post the content of japanese.txt that produces the error?

[olaf.meeuwissen@…]

Running Trac 1.0.1 with the 3.0.0dev version of the plugin. I downloaded the zip via the wiki page, so I don't have the exact version number handy. The zip file name included a 134546, so I guess I'm at or after r12030.

The only other plugin I have enabled at the moment is TracAccountManager 0.4.3. Babel is at 0.9.6 and our Trac environment has no other encoding issues (that I'm aware of). Subversion is 1.6.17 (r1128011). File I'm trying to include is attached.

Not sure if it matters but the server itself does not have any localization support installed (Debian wheezy, C system locale, no locales package, all upgrades applied as of writing).

[olaf.meeuwissen@…]

File I'm trying to include that exhibits the reported behaviour

[Ryan J Ollos]

I can reproduce with Genshi 0.7.0, but not 0.6.1. Are you running Genshi 0.7.0?

I'm not sure how to fix the issue, but I'll continue investigating.

[olaf.meeuwissen@…]

Replying to rjollos:

I can reproduce with Genshi 0.7.0, but not 0.6.1. Are you running Genshi 0.7.0?

Yes. FWIW, the instance runs in a Python virtual env and the "About Trac" page has:

Package	Version
Trac	1.0.1
Babel	0.9.6
Genshi	0.7 (without speedups)
mod_wsgi	3.3 (WSGIProcessGroup project-name WSGIApplicationGroup localhost:8080|/project-name)
pysqlite	2.6.3
Python	2.6.8 (unknown, Jan 26 2013, 15:02:25) [GCC 4.7.2]
setuptools	0.6c11
SQLite	3.7.13
Subversion	1.6.17 (r1128011)
jQuery	1.7.2

I'm not sure how to fix the issue, but I'll continue investigating.

Thanks!

[Ryan J Ollos]

I would suggest downgrading for the time being, to Genshi 0.6.1, if you wish to use this plugin. I spent a few hours trying to fix the issue, but I am not sure how to fix it yet, and it could be a while before I can devote more time to it.

[olaf.meeuwissen@…]

Thanks for your efforts so far and the feedback but I'm not sure that I can roll the affected Trac instance back to Genshi 0.6.1 any time soon, if at all.

[Ryan J Ollos]

It seems to work following the changes in #12422.

[anonymous]

Thanks! The file is included without error. Now I just need to figure out why the file gets stuck in a <pre> block despite its MIME property but that's another issue.