Opened 12 years ago
Last modified 7 years ago
#10188 reopened defect
simple import failed
Reported by: | falkb | Owned by: | François Granade |
---|---|---|---|
Priority: | normal | Component: | TicketImportPlugin |
Severity: | major | Keywords: | |
Cc: | Jun Omae | Trac Release: | 0.12 |
Description
The file is the attached .xls one. The callstack here is:
Traceback (most recent call last): File "build\bdist.win32\egg\trac\web\api.py", line 440, in send_error data, 'text/html') File "build\bdist.win32\egg\trac\web\chrome.py", line 868, in render_template stream.render(method, doctype=doctype, out=buffer) File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 183, in render return encode(generator, method=method, encoding=encoding, out=out) File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 58, in encode for chunk in iterator: File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 339, in __call__ for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 826, in __call__ for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 670, in __call__ for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 771, in __call__ for kind, data, pos in chain(stream, [(None, None, None)]): File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 586, in __call__ for ev in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build\bdist.win32\egg\trac\web\chrome.py", line 981, in _strip_accesskeys for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build\bdist.win32\egg\trac\web\chrome.py", line 970, in _generate for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match ctxt, start=idx + 1, **vars): File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match ctxt, start=idx + 1, **vars): File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 327, in _match for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate subevent = next() File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip event = next() File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten for kind, data, pos in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate subevent = next() File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include for event in stream: File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip event = next() File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 565, in _flatten result = _eval_expr(data, ctxt, vars) File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 277, in _eval_expr retval = expr.evaluate(ctxt) File "build/bdist.macosx-10.6-i386/egg/genshi/template/eval.py", line 178, in evaluate return eval(self.code, _globals, {'__data__': data}) File "d:/trac/test_devel\.egg-cache\Trac-0.12.2-py2.5.egg-tmp\trac\templates\error.html", line 165, in <Expression u'shorten_line(repr(value))'> <td><code>${shorten_line(repr(value))}</code></td> File "build\bdist.win32\egg\mastertickets\model.py", line 118, in __repr__ (self.tkt.id, l(getattr(self, 'blocking', [])), l(getattr(self, 'blocked_by', []))) File "build\bdist.win32\egg\mastertickets\model.py", line 115, in l return '[%s]'%','.join(arr2) TypeError: sequence item 0: expected string, int found
It's Trac-0.12.2 on Windows, TicketImportPlugin is the latest from SVN, TracMasterTickets-3.0.2-py2.5.egg is the latest available.
Attachments (3)
Change History (35)
Changed 12 years ago by
Changed 12 years ago by
Attachment: | to-unicode-if-custom-field-r11799.diff added |
---|
comment:1 Changed 12 years ago by
comment:2 follow-up: 3 Changed 12 years ago by
No, sorry, the patch didn't help. The callstack remains the same.
comment:3 Changed 12 years ago by
Sorry. I've misunderstood.
It seems that the issue has been fixed in https://github.com/coderanger/trac-mastertickets/commit/634d8784aed8f5a77cb3ffc470241bdab253c829. Could you please try the latest source from https://github.com/coderanger/trac-mastertickets/zipball/master?
comment:4 Changed 12 years ago by
Previously, I used masterticket version coderanger-trac-mastertickets-42b59b4
. Now I tried your recommended coderanger-trac-mastertickets-34803ad
. Your patch of 1 is still active.
The error on importing bug.xls turns to:
IntegrityError: columns ticket, time, field are not unique
with callstack:
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request dispatcher.dispatch(req) Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch resp = chosen_handler.process_request(req) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request encoding=req.session['importer.encoding']) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime)) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process processor.end_process_row() Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 138, in end_process_row self.ticket.insert(when=self._tickettime(), db=self.db) Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert listener.ticket_created(self) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created self.ticket_changed(tkt, '', tkt['reporter'], {}) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed links.save(author, comment, tkt.time_changed, db) Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save (n, when_ts, author, field, old_value, new_value)) Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute return self.cursor.execute(sql_escape_percent(sql), args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute result = PyFormatCursor.execute(self, *args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute args or []) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error return function(self, *args, **kwargs)
comment:5 Changed 12 years ago by
I think I know what the problem is... kind of my fault, I modified the patch when applying - I set the modification date when adding the master ticket info to the ticket, to be the same as the modification date of the first import... give me a few hours to fix it, I'm quite busy currently, but I'll get to it today.
comment:7 Changed 12 years ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
ok I've implemented a workaround in [11805]. In fact, it's not my fault, it's a bug in MasterTicketPlugin, and I have to work around it... I'll enter it...
comment:9 Changed 12 years ago by
Thanks. Unfortunately, this working week is over, and I wont be able to test it for some time now...
comment:10 Changed 12 years ago by
Resolution: | fixed |
---|---|
Status: | closed → reopened |
Now I was able to work further on this problem. The latest SVN version still doesn't work for that attached bug.xls
There's still
IntegrityError: columns ticket, time, field are not unique
The callstack is slightly different to the one of 4
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request dispatcher.dispatch(req) Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch resp = chosen_handler.process_request(req) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request encoding=req.session['importer.encoding']) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime)) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process processor.end_process_row() Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row self.ticket.insert(when=self._tickettime(), db=self.db) Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert listener.ticket_created(self) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created self.ticket_changed(tkt, '', tkt['reporter'], {}) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed links.save(author, comment, tkt.time_changed, db) Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 73, in save (n, when_ts, author, field, old_value, new_value)) Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute return self.cursor.execute(sql_escape_percent(sql), args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute result = PyFormatCursor.execute(self, *args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute args or []) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error return function(self, *args, **kwargs)
comment:11 follow-up: 12 Changed 12 years ago by
I've trried to reproduce your problem, but to no avail
What version of Trac are you running ? what version of the MasterTickets plugin ?
thanks
comment:12 Changed 12 years ago by
Replying to farialima:
What version of Trac are you running ?
Bitnami 0.12.2, Apache, Windows
what version of the MasterTickets plugin ?
coderanger-trac-mastertickets-34803ad
comment:13 follow-up: 14 Changed 12 years ago by
Well, unfortunately, there is no SVN version of the mastertickets plugin that can be checked for local modifications... so I compared again a newly downloaded 'coderanger-trac-mastertickets-34803ad' from web against my local copy of 'coderanger-trac-mastertickets-34803ad' and found out somewhen ago I obviously applied a small patch in def validate_ticket
in api.py:135. Maybe that's the reason?:
# Check that ticket does not have itself as a blocker - if id in links.blocking | links.blocked_by: - yield 'blocked_by', 'This ticket is blocking itself' + if id.isdecimal() and int(id) in links.blocking: + yield 'blocking', 'This ticket is blocking itself' + return + + if id.isdecimal() and int(id) in links.blocked_by: + yield 'blocked_by', 'This ticket is blocked by itself' return
I'll retry bug.xls
comment:14 follow-up: 15 Changed 12 years ago by
Replying to falkb:
I'll retry bug.xls
No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch):
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request dispatcher.dispatch(req) Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch resp = chosen_handler.process_request(req) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request encoding=req.session['importer.encoding']) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime)) Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process processor.end_process_row() Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row self.ticket.insert(when=self._tickettime(), db=self.db) Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert listener.ticket_created(self) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created self.ticket_changed(tkt, '', tkt['reporter'], {}) Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed links.save(author, comment, tkt.time_changed, db) Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save (n, when_ts, author, field, old_value, new_value)) Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute return self.cursor.execute(sql_escape_percent(sql), args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute result = PyFormatCursor.execute(self, *args) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute args or []) Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error return function(self, *args, **kwargs)
Systeminformationen:
User Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1
Trac 0.12.2 Babel 0.9.5 Bitten 0.7dev-r1008 FullBlog 0.1.1-r9430 Genshi 0.6 mod_python 3.3.1 pysqlite 2.5.5 Python 2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)] RPC 1.1.0 setuptools 0.6c11 SQLite 3.6.11 Subversion 1.6.6 (r40053) jQuery 1.4.2
comment:15 Changed 12 years ago by
Replying to falkb:
Replying to falkb:
I'll retry bug.xls
No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch)
and the original version of coderanger-trac-mastertickets-34803ad
comment:17 Changed 12 years ago by
I'm maintaining the MasterTicketsPlugin since very recently, and I acknowledge the defect reported in #10194. I've been able to reproduce it several times and hope to have the issue fixed in the next version.
comment:18 Changed 12 years ago by
rjollos, once you said you have got the fix almost ready at your machine and just need to commit then. Is there hope to get it somehow these days? To fix this issue here is quite important...
comment:19 Changed 12 years ago by
Unfortunately I lost that, and quite a bit of other work that I had in local Git repositories, when my virtual machine stopped booting. I intend to recreate the patches, and it is fairly high on my priority list, but I can't say for sure when it will be ready.
comment:20 Changed 12 years ago by
Should be fixed in MasterTicketsPlugin 3.0.4. Please report back if you have a chance to test.
comment:21 Changed 12 years ago by
Still has problems as reported in comment:ticket:10194:10. I used attachment:bug.xls for my test but it fails with that reported callstack.
comment:22 Changed 12 years ago by
This might be fixed in [12950]. I'm not very happy with the solution, but please give it a try and let me know if it is working for you now.
comment:23 follow-up: 24 Changed 12 years ago by
Now using todays latest SVN version of TH's MasterTicketsPlugin (r12950) and TicketImportPlugin (r12819) I tested attachment:bug.xls and the import of it works fine for it now => Thanks a lot!
Though trying a more complex .xls file I run into this integrity error:
File "build/bdist.win32/egg/trac/web/main.py", line 497, in _dispatch_request dispatcher.dispatch(req) File "build/bdist.win32/egg/trac/web/main.py", line 214, in dispatch resp = chosen_handler.process_request(req) File "build/bdist.win32/egg/talm_importer/importer.py", line 86, in process_request encoding=req.session['importer.encoding']) File "build/bdist.win32/egg/talm_importer/importer.py", line 130, in _do_import return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime)) File "build/bdist.win32/egg/talm_importer/importer.py", line 417, in _process processor.process_relativeticket_fields(relativeticketvalues, lowercaserelativeticketfields) File "build/bdist.win32/egg/talm_importer/processors.py", line 237, in process_relativeticket_fields self._save_ticket(ticket, with_comment=False) File "build/bdist.win32/egg/talm_importer/processors.py", line 113, in _save_ticket db=self.db) File "build/bdist.win32/egg/trac/ticket/model.py", line 351, in save_changes self[name])) File "build/bdist.win32/egg/trac/db/util.py", line 121, in execute cursor.execute(query, params) File "build/bdist.win32/egg/trac/db/util.py", line 65, in execute return self.cursor.execute(sql_escape_percent(sql), args) File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 78, in execute result = PyFormatCursor.execute(self, *args) File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 56, in execute args or []) File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error return function(self, *args, **kwargs)
comment:24 follow-up: 25 Changed 12 years ago by
Replying to falkb:
[...] Though trying a more complex .xls file I run into this integrity error:
Could you attach that file?
Changed 12 years ago by
Attachment: | importtest2.xls added |
---|
more complex file waking up an integrity bug
comment:25 follow-up: 26 Changed 12 years ago by
Replying to rjollos:
Replying to falkb:
[...] Though trying a more complex .xls file I run into this integrity error:
Could you attach that file?
I had to decorate the original cell content a bit (confidential data), uploaded it as attachment:importtest2.xls, but I'm still able to run into the same callstack with that new file. Although a bit of the import work seems to be done because the timeline says:
10:50 AM Tickets #2747,2910,4265,4266,4272 batch updated by falkb blocking changed
comment:26 Changed 12 years ago by
Replying to falkb:
... the timeline says:
10:50 AM Tickets #2747,2910,4265,4266,4272 batch updated by falkb blocking changed
I wonder why the timeline does not report the creation of 4265,4266 and 4272. The latest ticket before the import was 4263.
comment:28 Changed 11 years ago by
I think that is a duplicate of #10194, not a ticketimportplugin issue.
comment:30 Changed 10 years ago by
I get this integrity error when I try to import tickets to be blocked by themselves (which, I agree is stupid, but when you have a batch upload you don't necessarily look at every ticket).
It would be nice if user errors this were somehow trapped...
comment:31 Changed 10 years ago by
I can also work around the problem if I import the blocking and blocked by columns separately, i.e. if I first upload with just columns id & blocking, and then with just id & blocked by.
Thanks for the report. If the value of custom fields is not unicode, the issue will occur.
Could you please try the patch, to-unicode-if-custom-field-r11799.diff? (I will add unit tests for it later.)