ticket.get appears to die on ticket custom fields

[greg@…]

When trying to use ticket.get() against my trac installation I get a TypeError exception from dump_struct in xmlrpclib.py due to the type of one of my custom fields not being string. They are, in fact type 'unicode' so I'm not really sure if they are wrong or the plugin is wrong but either way it doesn't seem to work. This was also posted to the trac mailing list.

Sorry to respond to my own post but I've been debugging this a bit and
have more of an idea of what is going on.

I played around with the exception handling in xmlrpclib.py. It turns
out the TypeError exception gets thrown on one of my custom ticket
fields. The field is defined in trac.ini as:

verified = checkbox
verified.label = Fix Verified
verified.value = 0
verified.order = 7

and the type is somehow unicode. I have a few other checkbox custom
fields and if they are processed in order then they are all fine. This
is the same field I had problems with in ticket #3939 so now I'm
wondering if there is just something dodgy with that field. 

Anyone have an idea of how to track this down?

Thanks,
Greg


On Sat, 2006-11-04 at 18:23 -0500, Greg Dickie wrote:
> Hi,
> 
>   Sorry in advance if this is a FAQ but I'm an xmlrpc virgin ;-). I want
> to use xmlrpc to create tickets a part of a merge review process but I
> can't even get the examples to work right ;-). This is trac 0.10 and
> using the example:
> 
> import xmlrpclib
> 
> server = xmlrpclib.ServerProxy("http://trac/Sledgehammer/xmlrpc")
> 
> multicall = xmlrpclib.MultiCall(server)
> for ticket in server.ticket.query("owner=greg"):
>     multicall.ticket.get(ticket)
> print map(str, multicall())
> 
> 
> gives this:
> 
> Traceback (most recent call last):
>   File "tt", line 8, in ?
>     print map(str, multicall())
>   File "/usr/lib/python2.4/xmlrpclib.py", line 945, in __call__
>     return
> MultiCallIterator(self.__server.system.multicall(marshalled_list))
>   File "/usr/lib/python2.4/xmlrpclib.py", line 1096, in __call__
>     return self.__send(self.__name, args)
>   File "/usr/lib/python2.4/xmlrpclib.py", line 1383, in __request
>     verbose=self.__verbose
>   File "/usr/lib/python2.4/xmlrpclib.py", line 1147, in request
>     return self._parse_response(h.getfile(), sock)
>   File "/usr/lib/python2.4/xmlrpclib.py", line 1286, in _parse_response
>     return u.close()
>   File "/usr/lib/python2.4/xmlrpclib.py", line 744, in close
>     raise Fault(**self._stack[0])
> xmlrpclib.Fault: <Fault 2: "'dictionary key must be string' while
> executing 'system.multicall()'">
> 
> 
> which seems to be to be a problem in the data being received.
> 
> Anyone know what I'm doing wrong?
> 
> Thanks,
> Greg
> 

Thanks, Greg

[Shun-ichi Goto]

I think you'd better to check trac.log to get detail of error ('dictionary key must be string') beacuse error is occured while executing on trac.

I tried your code but I cannot reproduced in my environment which has selection custom field.

[anonymous]

Thanks for the suggestion. Note the traceback is slightly different because I changed the exception to tell me which field had the problem but its the same exception. From trac.log on the server I get:

2006-11-21 07:09:28,115 Trac[web_ui] ERROR: Traceback (most recent call last):
  File "build/bdist.linux-i686/egg/tracrpc/web_ui.py", line 58, in process_reque
st
  File "/usr/lib/python2.3/xmlrpclib.py", line 969, in dumps
    data = m.dumps(params)
  File "/usr/lib/python2.3/xmlrpclib.py", line 604, in dumps
    dump(v, write)
  File "/usr/lib/python2.3/xmlrpclib.py", line 616, in __dump
    f(self, value, write)
  File "/usr/lib/python2.3/xmlrpclib.py", line 676, in dump_array
    dump(v, write)
  File "/usr/lib/python2.3/xmlrpclib.py", line 616, in __dump
    f(self, value, write)
  File "/usr/lib/python2.3/xmlrpclib.py", line 699, in dump_struct
    raise NameError,  k
NameError: verified

[Shun-ichi Goto]

Yet I cannot reproduce your error although I've added same custom field 'verified' on trac 0.10.2 (on windows and python 2.3.5). I prefer to check (by logging, etc.) type of object in variable 'k' to ensure it is string or not when trapped before raise NameError (by your debug code, I guess) becaues the error "dictionary key must be string" is raised when 'k' is nether StringType nor UnicodeType (in xmlrpclib.dump_struct()).

[anonymous]

Thank you for looking at this. I have verified the type and it is, in fact, unicode but xmlrpclib only checks for string, if it check for unicode it would be fine. See below (with my debug code in it):

    def dump_struct(self, value, write, escape=escape):
        i = id(value)
        if self.memo.has_key(i):
            raise TypeError, "cannot marshal recursive dictionaries"
        self.memo[i] = None
        dump = self.__dump
        write("<value><struct>\n")
        for k in value.keys():
            write("<member>\n")
            try:
              raise NameError,  type(k)
            except NameError:
              print k
            if type(k) is not StringType:
              try:
                  raise TypeError, "dictionary key must be string"
              except TypeError:
                raise NameError,  k
                raise
            write("<name>%s</name>\n" % escape(k))
            dump(value[k], write)
            write("</member>\n")
        write("</struct></value>\n")
        del self.memo[i]
    dispatch[DictType] = dump_struct

This is python 2.3.4-14.3 on CentOS 4U4

[anonymous]

Just for fun, I added a check in for UnicodeType in xmlrpclib.py and now it seems to work. So I suppose the bug is really in xmlrpclib but any idea why custom fields are unicode but regular fields are string?

[Shun-ichi Goto]

Replying to anonymous:

This is python 2.3.4-14.3 on CentOS 4U4

dump_struct() of python 2.3.5 is:

    def dump_struct(self, value, write, escape=escape):
        i = id(value)
        if self.memo.has_key(i):
            raise TypeError, "cannot marshal recursive dictionaries"
        self.memo[i] = None
        dump = self.__dump
        write("<value><struct>\n")
        for k, v in value.items():
            write("<member>\n")
            if type(k) is not StringType:
                if unicode and type(k) is UnicodeType:
                    k = k.encode(self.encoding)
                else:
                    raise TypeError, "dictionary key must be string"
            write("<name>%s</name>\n" % escape(k))
            dump(v, write)
            write("</member>\n")
        write("</struct></value>\n")
        del self.memo[i]

It seems there is a difference in unicode consideration between 2.3.4 and 2.3.5. Thus, easiest workaround is modifying xmlrpclib.py like above or upgrade to 2.3.5 if possible. It seems a bug of xmlrpclib.

The reason of name of non-custom fields are not unicode is field name literals in TicketSystem.get_ticket_fields() are not unicode string. On the other hand, strings read from trac.ini are all unicode. It is the difference, I guess.

Following patch might be help if we make workaround in xmlrpcplugin to support python 2.3.4 or erlier.

ticket.py

@@ -67 +67 @@
     def get(self, req, id):
         """ Fetch a ticket. Returns [id, time_created, time_changed, attributes]. """
         t = model.Ticket(self.env, id)
-        return (t.id, t.time_created, t.time_changed, t.values)
+        values = {}
+        for k, v in t.values.items():
+            import sys
+            if sys.version_info[:3] < (2,3,5) and isinstance(k, unicode):
+                k = k.encode('utf-8') # workaround for old xmlrpclib
+            values[k] = v
+        return (t.id, t.time_created, t.time_changed, values)
 
     def create(self, req, summary, description, attributes = {}):
         """ Create a new ticket, returning the ticket ID. """

[greg@…]

Indeed it seems to be a problem in the python distribution however since I guess alot of people will want to stay with their distribution's currently "blessed" version its probably better to fix the plugin code. Your patch above is great but that is only for the get method. Do you know if there are other methods that will have the same issue? (eg: trying to set a custom field).

Thanks again for looking at this.

[Shun-ichi Goto]

Replying to greg@max-t.com:

Do you know if there are other methods that will have the same issue? (eg: trying to set a custom field).

I cannot test to find good or not because I don't have 2.3.4 environment (windows and debian). Please test for other methods on your environment then report here.

[rob.sharp@…]

I have patched the plugin as mentioned above, looks like it works well!

[osimons]

#1718 closed as duplicate.

[Thijs Triemstra]

python 2.3 is very old and proably shouldn't be supported anymore?

[osimons]

Problem is way too old... Whatever the problem was it is working now with newer versions. Just upgrade everything.