TocMacro stopped working with trac 1.0.1

[Jens Langner]

I recently recognized that the TocMacro somehow seem to have stopped working with my installed copy of trac 1.0.1. Here the following error is returned instead when a page is layouted which uses the TOC() macro:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 85: ordinal not in range(128)

The trac.log shows the following entries if that happens:

2013-06-02 13:16:16,974 Trac[formatter] ERROR: Macro TOC(titleindex, heading=FAQ Contents, notitle, depth=2, FAQ/General Questions, FAQ/Installing YAM, FAQ/Using YAM, FAQ/ARexx Scripting, FAQ/Debugging) failed:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 765, in _macro_formatter
    return macro.ensure_inline(macro.process(args))
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 356, in process
    text = self.processor(text)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 343, in _macro_processor
    text)
  File "build/bdist.linux-x86_64/egg/tractoc/macro.py", line 193, in expand_macro
    params['min_depth'] < 2)
  File "build/bdist.linux-x86_64/egg/tractoc/macro.py", line 215, in _render_title_index
    fmt.format(page_text, NullOut())
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1423, in format
    Formatter.format(self, text)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1283, in format
    result = re.sub(self.wikiparser.rules, self.replace, line)
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1201, in replace
    replacement = self.handle_match(fullmatch)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1197, in handle_match
    return internal_handler(match, fullmatch)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 579, in _shrefbr_formatter
    self._make_link(ns, target, match, match, fullmatch)
  File "/usr/lib64/python2.7/site-packages/genshi/builder.py", line 107, in __str__
    return str(self.generate())
UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 85: ordinal not in range(128)

If you want to have a look at the corresponding page which uses this macro have a look at:

​https://yam.ch/wiki/FAQ

There you should be able to see the error in question.

[Ryan J Ollos]

You are running Genshi 0.7? We've seen this problem in #10989 and #11145.

[Jens Langner]

yes, I am using genshi 0.7

[Ryan J Ollos]

The problem may not be with the TocMacro. Could you check if there is a traceback in the logs that results from accessing ​https://yam.ch/wiki/FAQ/Using%20YAM?

[Ryan J Ollos]

Note also,

• On the ​AllInOne page, the wiki in the included page FAQ/Using YAM is not formatted to HTML.
• When trying to ​export to PDF a similar error occurs, again in Formatter.format:

How to Reproduce

While doing a GET operation on /wiki/FAQ/Using YAM, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'format': u'pdf', 'page': u'FAQ/Using YAM'}

User agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22

System Information

System information not available

Enabled Plugins

Plugin information not available

Python Traceback

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/trac/web/main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib64/python2.7/site-packages/trac/web/main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/web_ui.py", line 177, in process_request
    format, versioned_page.name)
  File "/usr/lib64/python2.7/site-packages/trac/mimeview/api.py", line 1015, in send_converted
    content, selector)
  File "/usr/lib64/python2.7/site-packages/trac/mimeview/api.py", line 691, in convert_content
    output = converter.convert_content(req, mimetype, content, ck)
  File "build/bdist.linux-x86_64/egg/pagetopdf/pagetopdf.py", line 22, in convert_content
    page = wiki_to_html(source, self.env, req).encode(codepage, 'replace')
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1572, in wiki_to_html
    Formatter(env, context).format(wikitext, out, escape_newlines)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1283, in format
    result = re.sub(self.wikiparser.rules, self.replace, line)
  File "/usr/lib64/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1201, in replace
    replacement = self.handle_match(fullmatch)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 1197, in handle_match
    return internal_handler(match, fullmatch)
  File "/usr/lib64/python2.7/site-packages/trac/wiki/formatter.py", line 579, in _shrefbr_formatter
    self._make_link(ns, target, match, match, fullmatch)
  File "/usr/lib64/python2.7/site-packages/genshi/builder.py", line 107, in __str__
    return str(self.generate())
UnicodeEncodeError: 'ascii' codec can't encode character u'\u200b' in position 85: ordinal not in range(128)

When moving the content into a development instance, I'm still unable to reproduce.

[Ryan J Ollos]

I couldn't reproduce because I was testing with Trac 1.0-stable. I was able to reproduce when switching to Trac 1.0.1.

The problem has been fixed in ​[t 11683]. It's too bad there isn't a ticket associated with that changeset.

As far as I can see, you have three options:

1. Downgrade to Genshi 6.1
2. Upgrade to Trac 1.0-stable > ​t:r11683.
3. Remove http from [wiki] safe_schema. The result will be that URLs aren't rendered as hyperlinks.

[Ryan J Ollos]

Note in comment:6, a new resolution: cantfix. Trac has that resolution, and bugs filed against t.e.o that are plugin issues are frequently closed as cantfix. It seems appropriate and provides a nice sense of symmetry to do the same here when the problem is with Trac.

hasienda: does this sound okay to you?

[Steffen Hoffmann]

Replying to rjollos:

Note in comment:6, a new resolution: cantfix. Trac has that resolution, and bugs filed against t.e.o that are plugin issues are frequently closed as cantfix. It seems appropriate and provides a nice sense of symmetry to do the same here when the problem is with Trac.

hasienda: does this sound okay to you?

I agree, even if occasions may be rare, where nothing could be done for a plugin to work around Trac core issues.

Few weeks ago I encountered some code, where I needed to replace '' with u'' myself, what made me think a bit more. Neither looked into the Genshi change nore heard much about about an upcoming Python2to3 effort by now, but this is definitely related, because one of the major changes is nothing-else-than-bytecode-or-unicode directive in Python3.

[Jens Langner]

Replying to rjollos:

The problem has been fixed in ​[t 11683]. It's too bad there isn't a ticket associated with that changeset.

As far as I can see, you have three options:

1. Downgrade to Genshi 6.1
2. Upgrade to Trac 1.0-stable > ​t:r11683.
3. Remove http from [wiki] safe_schema. The result will be that URLs aren't rendered as hyperlinks.

Please note, that I took the 4. possibility in simply applying the fix to formatter.py in my trac installation which is pointed out in ​t:r11683 :)

Now everything seems to work fine and the problem doesn't occurr anymore. I guess, its time to see 1.0.2 being released soon...