API docs don't display

[Ryan J Ollos]

When on the Plugin Registry page (/developer/plugins) and trying to view the documentation for an interface, the documentation does not display and the following error is seen in the logs:

09:02:10 PM Trac[main] ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "/home/user/Workspace/tracdev/teo-rjollos.git/trac/web/main.py", line 543, in _dispatch_request
    dispatcher.dispatch(req)
  File "/home/user/Workspace/tracdev/teo-rjollos.git/trac/web/main.py", line 223, in dispatch
    resp = chosen_handler.process_request(req)
  File "/home/user/Workspace/trachacks.git/tracdeveloperplugin/trunk/tracdeveloper/apidoc.py", line 44, in process_request
    'doc': formatter(req, inspect.getdoc(obj)),
  File "/home/user/Workspace/trachacks.git/tracdeveloperplugin/trunk/tracdeveloper/apidoc.py", line 65, in _format_default
    return linebreaks(text)
  File "/home/user/Workspace/trachacks.git/tracdeveloperplugin/trunk/tracdeveloper/util.py", line 14, in linebreaks
    return HTML(''.join(paras))
  File "/home/user/Workspace/tracdev/local/lib/python2.7/site-packages/Genshi-0.7-py2.7-linux-x86_64.egg/genshi/input.py", line 442, in HTML
    return Stream(list(HTMLParser(BytesIO(text), encoding=encoding)))
  File "/home/user/Workspace/tracdev/local/lib/python2.7/site-packages/Genshi-0.7-py2.7-linux-x86_64.egg/genshi/core.py", line 273, in _ensure
    event = stream.next()
  File "/home/user/Workspace/tracdev/local/lib/python2.7/site-packages/Genshi-0.7-py2.7-linux-x86_64.egg/genshi/input.py", line 449, in _coalesce
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/home/user/Workspace/tracdev/local/lib/python2.7/site-packages/Genshi-0.7-py2.7-linux-x86_64.egg/genshi/input.py", line 337, in _generate
    raise UnicodeError("source returned bytes, but no encoding specified")
UnicodeError: source returned bytes, but no encoding specified

Issue is seen in 1.0.2dev and 1.1.2dev. I haven't tested with other versions.

[Ryan J Ollos]

Side note: there is an unresolved reference here: tracdeveloperplugin/trunk/tracdeveloper/apidoc.py@3230:62#L51​.

[Ryan J Ollos]

The issue occurs with Genshi 0.7. The following patch fixes the issue, but I am unsure as to whether it is the best way:

tracdeveloperplugin/trunk/tracdeveloper/apidoc.py

@@ -46 +46 @@
         }
         output = Chrome(self.env).render_template(req, 'developer/apidoc.html',
                                                   data, fragment=True)
-        req.send(output.render('xhtml'), 'text/html')
+        req.send(output.render('xhtml').encode('utf-8'), 'text/html')
 
     # Internal methods
     def _get_formatter(self, module):

tracdeveloperplugin/trunk/tracdeveloper/util.py

@@ -11 +11 @@
     value = re.sub(r'\r\n|\r|\n', '\n', value) # normalize newlines
     paras = re.split('\n{2,}', value)
     paras = ['<p>%s</p>' % p.strip().replace('\n', '<br />') for p in paras]
-    return HTML(''.join(paras))
+    return HTML(''.join(paras), encoding='utf-8')

[Jun Omae]

Replying to rjollos:

The issue occurs with Genshi 0.7. The following patch fixes the issue, but I am unsure as to whether it is the best way:

I think it would be simple to return template and data from process_request method rather than call directly render_template.

tracdeveloper/apidoc.py

@@ -44 +44 @@
             'doc': formatter(req, inspect.getdoc(obj)),
             'methods': self._get_methods(req, formatter, obj)
         }
-        output = Chrome(self.env).render_template(req, 'developer/apidoc.html',
-                                                  data, fragment=True)
-        req.send(output.render('xhtml'), 'text/html')
+        return 'developer/apidoc.html', data, 'text/html'
 
     # Internal methods
     def _get_formatter(self, module):

[Jun Omae]

Also, linebreaks method has an XSS issue. For example, documentation of trac.mimeview.api:IHTMLPreviewRenderer has <object> and <img>. /developer/doc/trac.mimeview.api:IHTMLPreviewRenderer page renders thoes text without escape.

The following patch works well on Genshi 0.6 and 0.7.

tracdeveloper/util.py

@@ -2 +2 @@
 
 import re
 
-from genshi import HTML
+from genshi.builder import tag
 
 def linebreaks(value):
     """Converts newlines in strings into <p> and <br />s."""
@@ -10 +10 @@
         return ''
     value = re.sub(r'\r\n|\r|\n', '\n', value) # normalize newlines
     paras = re.split('\n{2,}', value)
-    paras = ['<p>%s</p>' % p.strip().replace('\n', '<br />') for p in paras]
-    return HTML(''.join(paras))
+    return tag(tag.p((line, tag.br) for line in para.splitlines())
+               for para in paras)

[Jun Omae]

Another thing: I think /developer/api should require TRAC_DEVELOP permission. Currently, anyone can let to import any objects using the feature.

tracdeveloper/apidoc.py

@@ -29 +29 @@
             return True
 
     def process_request(self, req):
+        req.perm.require('TRAC_DEVELOP')
         modname, attrname = str(req.args['name']).split(':')
         try:
             module = __import__(modname, {}, {}, filter(None, [attrname]))

[Olemis Lang]

Overall , I agree with all the suggestions provided in comment:6 , comment:5 . I'm still not sure about comment:4 , but at first sight it looks fine to me . I'll take a look into this issue in the next few days .

[Jun Omae]

Another thing: I think /developer/api should require TRAC_DEVELOP permission. Currently, anyone can let to import any objects using the feature.

Reconsidered about that, I think it would safe to use sys.modules rather than __import__. See attachment:t11907.diff​.

[Ryan J Ollos]

In 14481:

TracDeveloper 0.3.0dev: Fix UnicodeEncode error rendering API docs

Fixes #11907.