UnicodeEncodeError: 'ascii' codec can't encode characters in position 9-10: ordinal not in range(128)

[manuel.neto]

How to Reproduce

While doing a GET operation on /admin/accounts/users, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'cat_id': u'accounts', 'panel_id': u'users', 'path_info': None}

User agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1

System Information

Trac	0.12.3
Trac	0.12.3
Agilo	0.9.8
Babel	0.9.6
CustomFieldAdmin	0.2.6
Docutils	0.8.1
Genshi	0.6
mod_wsgi	3.3 (WSGIProcessGroup WSGIApplicationGroup trac2.atualtec.com|)
psycopg2	2.2.1
Pygments	1.4
Python	2.6.6 (r266:84292, Dec 26 2010, 22:48:11) [GCC 4.4.5]
Python	2.6.6 (r266:84292, Dec 26 2010, 22:48:11) [GCC 4.4.5]
pytz	2011n
pytz	2011n
RPC	1.1.2
setuptools	0.6c11
setuptools	0.6c11
Subversion	1.6.12 (r955767)
jQuery	1.4.4

Enabled Plugins

agilo	0.9.8
AutocompleteUsers	0.4.2
AutoUpgrade	0.1
DiskUsagePlugin	0.2
IniAdmin	0.2
LdapPlugin	0.6.0dev
siteupload	0.11
TracAccountManager	0.3.2
TracActiveDirectoryAuth	0.2.2
TracAnnouncer	0.11.1
TracCustomFieldAdmin	0.2.6
TracDragDrop	0.12.0.5
TracRevtreePlugin	0.6.4dev
TracWikiPrintPlugin	1.8.4
TracXMLRPC	1.1.2

Python Traceback

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/web/main.py", line 243, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3-py2.6.egg/trac/admin/web_ui.py", line 117, in process_request
    path_info)
  File "build/bdist.linux-x86_64/egg/acct_mgr/admin.py", line 137, in render_admin_panel
    return self._do_users(req)
  File "build/bdist.linux-x86_64/egg/acct_mgr/admin.py", line 313, in _do_users
    for username in acctmgr.get_users():
  File "build/bdist.linux-x86_64/egg/acct_mgr/api.py", line 184, in get_users
    users.extend(store.get_users())
  File "build/bdist.linux-x86_64/egg/tracext/adauth/auth.py", line 53, in get_users
    'proxyAddresses', 'displayName'])
  File "/usr/lib/python2.6/dist-packages/ldap/ldapobject.py", line 502, in search_s
    return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
  File "/usr/lib/python2.6/dist-packages/ldap/ldapobject.py", line 495, in search_ext_s
    msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
  File "/usr/lib/python2.6/dist-packages/ldap/ldapobject.py", line 491, in search_ext
    timeout,sizelimit,
  File "/usr/lib/python2.6/dist-packages/ldap/ldapobject.py", line 96, in _ldap_call
    result = func(*args,**kwargs)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 9-10: ordinal not in range(128)

[hasienda]

At least one point down this traceback there is wrong handling of non-ASCII/Unicode characters. But it's not obvious where yet. We need more information and look at all code involved to find and correct it.

[hasienda]

Out fo curiosity: Is your LDAP backend guaranteed to properly handle Unicode characters?

[hasienda]

I can't do more without feedback. Please follow-up on my previous questions.

[hasienda]

Just found out a bit more about the source of this issue, that makes me believe, that sandinak might be in a better position to fix it than me.

Rationale: The last caller is not AccountManagerPlugin, it's ActiveDirectoryAuthPlugin (today known as DirectoryAuthPlugin), a password store for AccountManagerPlugin that provides authentication and groups from Lightweight Directory Access Protocol (LDAP) enabled service.

Maybe this has already been fixed meanwhile, and upgrading your ActiveDirectoryAuthPlugin to a recent version of it's successor would help?

[sima.baymani@…]

I have Trac 1.0 installed on Ubuntu, using DirectoryAuthPlugin r12025 (HEAD on trunk at this date). I can verify that this problem still exists for users with non-ASCII characters in their full names.

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/api.py", line 502, in send_error
    data, 'text/html')
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/chrome.py", line 955, in render_template
    message = req.session.pop('chrome.%s.%d' % (type_, i))
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/api.py", line 304, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/main.py", line 268, in _get_session
    return Session(self.env, req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/session.py", line 200, in __init__
    if req.authname == 'anonymous':
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/api.py", line 304, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/lib/python2.6/dist-packages/Trac-1.0-py2.6.egg/trac/web/main.py", line 135, in authenticate
    authname = authenticator.authenticate(req)
  File "/usr/local/lib/python2.6/dist-packages/TracAccountManager-0.4-py2.6.egg/acct_mgr/util.py", line 45, in wrap
    return func(self, *args, **kwds)
  File "/usr/local/lib/python2.6/dist-packages/TracAccountManager-0.4-py2.6.egg/acct_mgr/web_ui.py", line 300, in authenticate
    user = self._remote_user(req)
  File "/usr/local/lib/python2.6/dist-packages/TracAccountManager-0.4-py2.6.egg/acct_mgr/web_ui.py", line 649, in _remote_user
    if acctmgr.check_password(user, password) == True:
  File "/usr/local/lib/python2.6/dist-packages/TracAccountManager-0.4-py2.6.egg/acct_mgr/api.py", line 259, in check_password
    valid = store.check_password(user, password)
  File "/usr/local/lib/python2.6/dist-packages/TracDirectoryAuth-0.5-py2.6.egg/tracext/adauth/auth.py", line 150, in check_password
    success = self._bind_dir(dn, password) or False
  File "/usr/local/lib/python2.6/dist-packages/TracDirectoryAuth-0.5-py2.6.egg/tracext/adauth/auth.py", line 256, in _bind_dir
    self.log.debug('_bind_dir: attempting specific bind to %s as %s' % (self.dir_uri, user_dn))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)

I have read a suggestion on the general problem,adding a 'u' to the format string - but this does not seem to help. (Search for "Django: How to fix the annoying UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)" as the spam filter prevented me from adding the link)

Using the same patch as in #10443 (str.encode('utf-8')) does not help either.

I'm at loss here and some users can't log in - any help or suggestion would be highly appreciated.

[sima.baymani@…]

Changing Trac release from 0.11 to 1.0, since it is there as well. I haven't had time to look much on it myself but it seems that some rework on the code may be needed on how strings are handled.

I'll pop in again if I find a solution.

[anonymous]

Researched and found ​http://blog.codekills.net/2008/05/01/encoding-and-decoding-text-in-python-%28or---i-didn%27t-ask-you-to-use-the-%27ascii%27-codec!-%29/

Which talks about this case. I don't have a Unicode DN to test with ;-) I have patched it for the next release .. I'd ask anonymoose to test .. but 'es not around.

Also checked any further printing of user_dn in the debug output .. none here.

[hasienda]

You might want to use a slightly more trac-ish markup for commit messages to help TicketCommitUpdater pick and add ticket comments like so:

(In [12719]) Fix UnicodeEncodeError?: 'ascii' codec can't encode characters in position 9-10: ordinal not in range(128), closes #10139.

Just use 'closes #<tkt_no>' (comment and resolve) or 'refs #<tkt_no>' (comment-only) to trigger actions. Added bonus are back-links to the changeset from as many tickets (at trac-hacks.org) as you like. Needless to say: Handle with care and double-check numbers because at least links from repository to tickets are immutable of course.

[rjollos]

Replying to hasienda:

You might want to use a slightly more trac-ish markup for commit messages to help TicketCommitUpdater pick and add ticket comments like so: [...]

Yes, I would definitely recommend what hasienda suggested! It's one of the best features we have on this now ancient site :)