simple import failed

[falkb]

The file is the attached .xls one. The callstack here is:

Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\api.py", line 440, in send_error
    data, 'text/html')
  File "build\bdist.win32\egg\trac\web\chrome.py", line 868, in render_template
    stream.render(method, doctype=doctype, out=buffer)
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 981, in _strip_accesskeys
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 970, in _generate
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "d:/trac/test_devel\.egg-cache\Trac-0.12.2-py2.5.egg-tmp\trac\templates\error.html", line 165, in <Expression u'shorten_line(repr(value))'>
    <td><code>${shorten_line(repr(value))}</code></td>
  File "build\bdist.win32\egg\mastertickets\model.py", line 118, in __repr__
    (self.tkt.id, l(getattr(self, 'blocking', [])), l(getattr(self, 'blocked_by', [])))
  File "build\bdist.win32\egg\mastertickets\model.py", line 115, in l
    return '[%s]'%','.join(arr2)
TypeError: sequence item 0: expected string, int found

It's Trac-0.12.2 on Windows, TicketImportPlugin is the latest from SVN, TracMasterTickets-3.0.2-py2.5.egg is the latest available.

[jun66j5]

Thanks for the report. If the value of custom fields is not unicode, the issue will occur.

Could you please try the patch, to-unicode-if-custom-field-r11799.diff​?
(I will add unit tests for it later.)

[falkb]

No, sorry, the patch didn't help. The callstack remains the same.

[jun66j5]

Sorry. I've misunderstood.

It seems that the issue has been fixed in ​https://github.com/coderanger/trac-mastertickets/commit/634d8784aed8f5a77cb3ffc470241bdab253c829. Could you please try the latest source from ​https://github.com/coderanger/trac-mastertickets/zipball/master?

[falkb]

Previously, I used masterticket version coderanger-trac-mastertickets-42b59b4 . Now I tried your recommended coderanger-trac-mastertickets-34803ad . Your patch of 1 is still active.

The error on importing bug.xls turns to:
IntegrityError: columns ticket, time, field are not unique

with callstack:

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 138, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

[farialima]

I think I know what the problem is... kind of my fault, I modified the patch when applying - I set the modification date when adding the master ticket info to the ticket, to be the same as the modification date of the first import... give me a few hours to fix it, I'm quite busy currently, but I'll get to it today.

[falkb]

Any news?

[farialima]

ok I've implemented a workaround in [11805]. In fact, it's not my fault, it's a bug in MasterTicketPlugin, and I have to work around it... I'll enter it...

[farialima]

Root issue entered as #10194

[falkb]

Thanks. Unfortunately, this working week is over, and I wont be able to test it for some time now...

[falkb]

Now I was able to work further on this problem. The latest SVN version still doesn't work for that attached bug.xls

There's still

IntegrityError: columns ticket, time, field are not unique

The callstack is slightly different to the one of 4

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 73, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

[farialima]

I've trried to reproduce your problem, but to no avail

What version of Trac are you running ? what version of the MasterTickets plugin ?

thanks

[falkb]

Replying to farialima:

What version of Trac are you running ?

Bitnami 0.12.2, Apache, Windows

what version of the MasterTickets plugin ?

coderanger-trac-mastertickets-34803ad

[falkb]

Well, unfortunately, there is no SVN version of the mastertickets plugin that can be checked for local modifications... so I compared again a newly downloaded 'coderanger-trac-mastertickets-34803ad' from web against my local copy of 'coderanger-trac-mastertickets-34803ad' and found out somewhen ago I obviously applied a small patch in def validate_ticket in api.py:135. Maybe that's the reason?:

        # Check that ticket does not have itself as a blocker 
-        if id in links.blocking | links.blocked_by:
-            yield 'blocked_by', 'This ticket is blocking itself'
+        if id.isdecimal() and int(id) in links.blocking:
+            yield 'blocking', 'This ticket is blocking itself'
+            return
+            
+        if id.isdecimal() and int(id) in links.blocked_by:
+            yield 'blocked_by', 'This ticket is blocked by itself'
            return

I'll retry bug.xls

[falkb]

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch):

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

Systeminformationen:

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1

Trac 	0.12.2
Babel 	0.9.5
Bitten 	0.7dev-r1008
FullBlog 	0.1.1-r9430
Genshi 	0.6
mod_python 	3.3.1
pysqlite 	2.5.5
Python 	2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
RPC 	1.1.0
setuptools 	0.6c11
SQLite 	3.6.11
Subversion 	1.6.6 (r40053)
jQuery	1.4.2

[falkb]

Replying to falkb:

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch)

and the original version of coderanger-trac-mastertickets-34803ad

[falkb]

farialima, do you still have an idea how to get further here?

[rjollos]

I'm maintaining the MasterTicketsPlugin since very recently, and I acknowledge the defect reported in #10194. I've been able to reproduce it several times and hope to have the issue fixed in the next version.

[falkb]

rjollos, once you said you have got the fix almost ready at your machine and just need to commit then. Is there hope to get it somehow these days? To fix this issue here is quite important...

[rjollos]

Unfortunately I lost that, and quite a bit of other work that I had in local Git repositories, when my virtual machine stopped booting. I intend to recreate the patches, and it is fairly high on my priority list, but I can't say for sure when it will be ready.

[rjollos]

Should be fixed in MasterTicketsPlugin 3.0.4. Please report back if you have a chance to test.

[falkb]

Still has problems as reported in comment:ticket:10194:10. I used attachment:bug.xls​ for my test but it fails with that reported callstack.

[rjollos]

This might be fixed in [12950]. I'm not very happy with the solution, but please give it a try and let me know if it is working for you now.

[falkb]

Now using todays latest SVN version of TH's MasterTicketsPlugin (r12950) and TicketImportPlugin (r12819) I tested attachment:bug.xls​ and the import of it works fine for it now => Thanks a lot!

Though trying a more complex .xls file I run into this integrity error:

File "build/bdist.win32/egg/trac/web/main.py", line 497, in _dispatch_request
  dispatcher.dispatch(req)
File "build/bdist.win32/egg/trac/web/main.py", line 214, in dispatch
  resp = chosen_handler.process_request(req)
File "build/bdist.win32/egg/talm_importer/importer.py", line 86, in process_request
  encoding=req.session['importer.encoding'])
File "build/bdist.win32/egg/talm_importer/importer.py", line 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
File "build/bdist.win32/egg/talm_importer/importer.py", line 417, in _process
  processor.process_relativeticket_fields(relativeticketvalues, lowercaserelativeticketfields)
File "build/bdist.win32/egg/talm_importer/processors.py", line 237, in process_relativeticket_fields
  self._save_ticket(ticket, with_comment=False)
File "build/bdist.win32/egg/talm_importer/processors.py", line 113, in _save_ticket
  db=self.db)
File "build/bdist.win32/egg/trac/ticket/model.py", line 351, in save_changes
  self[name]))
File "build/bdist.win32/egg/trac/db/util.py", line 121, in execute
  cursor.execute(query, params)
File "build/bdist.win32/egg/trac/db/util.py", line 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 78, in execute
  result = PyFormatCursor.execute(self, *args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 56, in execute
  args or [])
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
  return function(self, *args, **kwargs)

[rjollos]

Replying to falkb:

[...]
Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

[falkb]

more complex file waking up an integrity bug

[falkb]

Replying to rjollos:

Replying to falkb:

[...]
Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

I had to decorate the original cell content a bit (confidential data), uploaded it as attachment:importtest2.xls​, but I'm still able to run into the same callstack with that new file. Although a bit of the import work seems to be done because the timeline says:

 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb
    blocking changed

[falkb]

Replying to falkb:

... the timeline says:

 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb
    blocking changed

I wonder why the timeline does not report the creation of 4265,​4266 and ​4272. The latest ticket before the import was 4263.