Error sending notifications for posts with unicode subjects

[luis@…]

When sending an email notification for posts with the subject written using unicode chars, I get the following error:

  File "/usr/local//lib/python2.3/site-packages/trac/web/main.py", line 387, in dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local//lib/python2.3/site-packages/trac/web/main.py", line 237, in dispatch 
    resp = chosen_handler.process_request(req)
  File "build/bdist.linux-i686/egg/tracdiscussion/core.py", line 72, in process_request
  File "build/bdist.linux-i686/egg/tracdiscussion/api.py", line 33, in render_discussion 
  File "build/bdist.linux-i686/egg/tracdiscussion/api.py", line 597, in _do_action
  File "build/bdist.linux-i686/egg/tracdiscussion/notification.py", line 91, in notify
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 216, in notify 
    Notify.notify(self, resid)
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 115, in notify
    self.send(torcpts, ccrcpts)
  File "build/bdist.linux-i686/egg/tracdiscussion/notification.py", line 130, in send 
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 360, in send
    self.add_headers(msg, headers);
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 236, in add_headers 
    msg[h] = self.encode_header(h, headers[h])
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 275, in encode_header
    return self.format_header(key, value)
  File "/usr/local//lib/python2.3/site-packages/trac/notification.py", line 225, in format_header 
    tmp = name.encode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 43: ordinal not in range(128)

I first posted this bug in the trac's trac, but was redirected here because this error doesn't happen when sending other notifications (from the ticket system for example).

For reference: ​http://trac.edgewall.org/ticket/4875

[ento]

A really quick patch. Works for me but I'm not sure if it's the correct way to fix this.

0.10/tracdiscussion/notification.py

@@ -88 +88 @@
         # Render body and send notification.
         subject = self.hdf.render('discussion-notify-subject.cs')
         self.env.log.debug(subject)
-        NotifyEmail.notify(self, id, subject)
+        NotifyEmail.notify(self, id, unicode(subject, 'utf-8'))
 
     def get_topic_id(self, forum_id, topic_id):
         return "%s-%s-%s" % (forum_id, topic_id, 0)

[Radek Bartoň]

I have suspicion that this error is duplicate of some later ticket that I desperatelly can't find. My memory is tricking me :-). Anyway, point of that ticket was that user has created PostgreSQL database with wrong encoding settings. Just changing that setting for all text attributes in discussion tables helped. Are you using PostgreSQL? I didn't find time to check this on PostgreSQL for me but I find out that problem is with reading from database. Text is stored as unicode but loaded with mallformed characters. I'll turn my focus on this right after I'll port DiscussionPlugin to 0.11.

[ento]

No, I'm using sqlite.

To reproduce this bug in my current environment setup,

1. Try creating a new topic with Japanese text (I only tried with both subject and body in Japanese)
   • sample: テスト
2. I get the same error as in the ticket description (at least the last few lines of the stack trace)

I just looked into the sqlite database; although there were several attempts to create topics before, I can only find topics that were created after I patched up the plugin. Notifications for ticket changes are working smoothly, with Japanese text in the subject and body.

Please take your time porting =) - I can live on with the makeshift patch for now.

[luis@…]

Sorry for the late reply but i've been a little busy this past week. The bug really appears when using sqlite.

If you check the reference, there I refer that changing a configuration relating the default encoding of python system wide corrects the problem. So this should be a conversion that's forgotten somewhere... But again... I'm no expert...

[osimons]

Problem confirmed for both 0.10 and current 0.11dev.

The first comment correctly identifies the location of the error - the subject need to be Unicode, and it isn't based on what I read from the 0.11dev traceback information.

Here is a 0.11dev patch that solves it - using Trac to_unicode() that is a bit smarter than the usual unicode(), and additionally I think the implementation means it will never throw an exception on usage. This function is also available for 0.10.

0.11/tracdiscussion/notification.py

@@ -4 +4 @@
 from trac.web.chrome import Chrome
 from trac.notification import NotifyEmail
 from trac.util import format_datetime
+from trac.util.text import to_unicode
 
 from trac.web.chrome import ITemplateProvider
 
@@ -79 +80 @@
           'subject' : subject, 'body' : body, 'link' : link}})
 
         # Render subject template and send notification.
-        subject = Chrome(self.env).render_template(context.req,
-          'discussion-notify-subject.txt', self.data, 'text/plain')
+        subject = to_unicode(Chrome(self.env).render_template(context.req,
+          'discussion-notify-subject.txt', self.data, 'text/plain'))
         NotifyEmail.notify(self, id, subject)
 
     def get_topic_id(self, forum_id, topic_id):

Please update for 0.10 and 0.11.

[Radek Bartoň]

Applied with r2837.