Modify

Opened 2 years ago

Last modified 11 months ago

#10188 reopened defect

simple import failed

Reported by: falkb Owned by: farialima
Priority: normal Component: TicketImportPlugin
Severity: major Keywords:
Cc: jun66j5 Trac Release: 0.12

Description

The file is the attached .xls one. The callstack here is:

Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\api.py", line 440, in send_error
    data, 'text/html')
  File "build\bdist.win32\egg\trac\web\chrome.py", line 868, in render_template
    stream.render(method, doctype=doctype, out=buffer)
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 981, in _strip_accesskeys
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 970, in _generate
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "d:/trac/test_devel\.egg-cache\Trac-0.12.2-py2.5.egg-tmp\trac\templates\error.html", line 165, in <Expression u'shorten_line(repr(value))'>
    <td><code>${shorten_line(repr(value))}</code></td>
  File "build\bdist.win32\egg\mastertickets\model.py", line 118, in __repr__
    (self.tkt.id, l(getattr(self, 'blocking', [])), l(getattr(self, 'blocked_by', [])))
  File "build\bdist.win32\egg\mastertickets\model.py", line 115, in l
    return '[%s]'%','.join(arr2)
TypeError: sequence item 0: expected string, int found

It's Trac-0.12.2 on Windows, TicketImportPlugin is the latest from SVN, TracMasterTickets-3.0.2-py2.5.egg is the latest available.

Attachments (3)

bug.xls (29.0 KB) - added by falkb 2 years ago.
to-unicode-if-custom-field-r11799.diff (1.3 KB) - added by jun66j5 2 years ago.
importtest2.xls (26.0 KB) - added by falkb 16 months ago.
more complex file waking up an integrity bug

Download all attachments as: .zip

Change History (31)

Changed 2 years ago by falkb

Changed 2 years ago by jun66j5

comment:1 Changed 2 years ago by jun66j5

Thanks for the report. If the value of custom fields is not unicode, the issue will occur.

Could you please try the patch, to-unicode-if-custom-field-r11799.diff?
(I will add unit tests for it later.)

comment:2 follow-up: Changed 2 years ago by falkb

No, sorry, the patch didn't help. The callstack remains the same.

comment:3 in reply to: ↑ 2 Changed 2 years ago by jun66j5

Sorry. I've misunderstood.

It seems that the issue has been fixed in https://github.com/coderanger/trac-mastertickets/commit/634d8784aed8f5a77cb3ffc470241bdab253c829. Could you please try the latest source from https://github.com/coderanger/trac-mastertickets/zipball/master?

comment:4 Changed 2 years ago by falkb

Previously, I used masterticket version coderanger-trac-mastertickets-42b59b4 . Now I tried your recommended coderanger-trac-mastertickets-34803ad . Your patch of 1 is still active.

The error on importing bug.xls turns to:
IntegrityError: columns ticket, time, field are not unique

with callstack:

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 138, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

comment:5 Changed 2 years ago by farialima

I think I know what the problem is... kind of my fault, I modified the patch when applying - I set the modification date when adding the master ticket info to the ticket, to be the same as the modification date of the first import... give me a few hours to fix it, I'm quite busy currently, but I'll get to it today.

comment:6 Changed 2 years ago by falkb

Any news?

comment:7 Changed 2 years ago by farialima

  • Resolution set to fixed
  • Status changed from new to closed

ok I've implemented a workaround in [11805]. In fact, it's not my fault, it's a bug in MasterTicketPlugin, and I have to work around it... I'll enter it...

comment:8 Changed 2 years ago by farialima

Root issue entered as #10194

comment:9 Changed 2 years ago by falkb

Thanks. Unfortunately, this working week is over, and I wont be able to test it for some time now...

comment:10 Changed 23 months ago by falkb

  • Resolution fixed deleted
  • Status changed from closed to reopened

Now I was able to work further on this problem. The latest SVN version still doesn't work for that attached bug.xls

There's still

IntegrityError: columns ticket, time, field are not unique

The callstack is slightly different to the one of 4

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 73, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

comment:11 follow-up: Changed 23 months ago by farialima

I've trried to reproduce your problem, but to no avail

What version of Trac are you running ? what version of the MasterTickets plugin ?

thanks

comment:12 in reply to: ↑ 11 Changed 23 months ago by falkb

Replying to farialima:

What version of Trac are you running ?

Bitnami 0.12.2, Apache, Windows

what version of the MasterTickets plugin ?

coderanger-trac-mastertickets-34803ad

comment:13 follow-up: Changed 23 months ago by falkb

Well, unfortunately, there is no SVN version of the mastertickets plugin that can be checked for local modifications... so I compared again a newly downloaded 'coderanger-trac-mastertickets-34803ad' from web against my local copy of 'coderanger-trac-mastertickets-34803ad' and found out somewhen ago I obviously applied a small patch in def validate_ticket in api.py:135. Maybe that's the reason?:

        # Check that ticket does not have itself as a blocker 
-        if id in links.blocking | links.blocked_by:
-            yield 'blocked_by', 'This ticket is blocking itself'
+        if id.isdecimal() and int(id) in links.blocking:
+            yield 'blocking', 'This ticket is blocking itself'
+            return
+            
+        if id.isdecimal() and int(id) in links.blocked_by:
+            yield 'blocked_by', 'This ticket is blocked by itself'
            return

I'll retry bug.xls

comment:14 in reply to: ↑ 13 ; follow-up: Changed 23 months ago by falkb

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch):

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

Systeminformationen:

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1

Trac 	0.12.2
Babel 	0.9.5
Bitten 	0.7dev-r1008
FullBlog 	0.1.1-r9430
Genshi 	0.6
mod_python 	3.3.1
pysqlite 	2.5.5
Python 	2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
RPC 	1.1.0
setuptools 	0.6c11
SQLite 	3.6.11
Subversion 	1.6.6 (r40053)
jQuery	1.4.2

comment:15 in reply to: ↑ 14 Changed 23 months ago by falkb

Replying to falkb:

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch)

and the original version of coderanger-trac-mastertickets-34803ad

comment:16 Changed 21 months ago by falkb

farialima, do you still have an idea how to get further here?

comment:17 Changed 21 months ago by rjollos

I'm maintaining the MasterTicketsPlugin since very recently, and I acknowledge the defect reported in #10194. I've been able to reproduce it several times and hope to have the issue fixed in the next version.

comment:18 Changed 17 months ago by falkb

rjollos, once you said you have got the fix almost ready at your machine and just need to commit then. Is there hope to get it somehow these days? To fix this issue here is quite important...

comment:19 Changed 17 months ago by rjollos

Unfortunately I lost that, and quite a bit of other work that I had in local Git repositories, when my virtual machine stopped booting. I intend to recreate the patches, and it is fairly high on my priority list, but I can't say for sure when it will be ready.

comment:20 Changed 17 months ago by rjollos

Should be fixed in MasterTicketsPlugin 3.0.4. Please report back if you have a chance to test.

comment:21 Changed 16 months ago by falkb

Still has problems as reported in comment:ticket:10194:10. I used attachment:bug.xls for my test but it fails with that reported callstack.

comment:22 Changed 16 months ago by rjollos

This might be fixed in [12950]. I'm not very happy with the solution, but please give it a try and let me know if it is working for you now.

comment:23 follow-up: Changed 16 months ago by falkb

Now using todays latest SVN version of TH's MasterTicketsPlugin (r12950) and TicketImportPlugin (r12819) I tested attachment:bug.xls and the import of it works fine for it now => Thanks a lot!

Though trying a more complex .xls file I run into this integrity error:

File "build/bdist.win32/egg/trac/web/main.py", line 497, in _dispatch_request
  dispatcher.dispatch(req)
File "build/bdist.win32/egg/trac/web/main.py", line 214, in dispatch
  resp = chosen_handler.process_request(req)
File "build/bdist.win32/egg/talm_importer/importer.py", line 86, in process_request
  encoding=req.session['importer.encoding'])
File "build/bdist.win32/egg/talm_importer/importer.py", line 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
File "build/bdist.win32/egg/talm_importer/importer.py", line 417, in _process
  processor.process_relativeticket_fields(relativeticketvalues, lowercaserelativeticketfields)
File "build/bdist.win32/egg/talm_importer/processors.py", line 237, in process_relativeticket_fields
  self._save_ticket(ticket, with_comment=False)
File "build/bdist.win32/egg/talm_importer/processors.py", line 113, in _save_ticket
  db=self.db)
File "build/bdist.win32/egg/trac/ticket/model.py", line 351, in save_changes
  self[name]))
File "build/bdist.win32/egg/trac/db/util.py", line 121, in execute
  cursor.execute(query, params)
File "build/bdist.win32/egg/trac/db/util.py", line 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 78, in execute
  result = PyFormatCursor.execute(self, *args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 56, in execute
  args or [])
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
  return function(self, *args, **kwargs)

comment:24 in reply to: ↑ 23 ; follow-up: Changed 16 months ago by rjollos

Replying to falkb:

[...]
Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

Changed 16 months ago by falkb

more complex file waking up an integrity bug

comment:25 in reply to: ↑ 24 ; follow-up: Changed 16 months ago by falkb

Replying to rjollos:

Replying to falkb:

[...]
Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

I had to decorate the original cell content a bit (confidential data), uploaded it as attachment:importtest2.xls, but I'm still able to run into the same callstack with that new file. Although a bit of the import work seems to be done because the timeline says:

 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb
    blocking changed

comment:26 in reply to: ↑ 25 Changed 16 months ago by falkb

Replying to falkb:

... the timeline says:

 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb
    blocking changed

I wonder why the timeline does not report the creation of 4265,​4266 and ​4272. The latest ticket before the import was 4263.

comment:27 Changed 11 months ago by falkb

Have you reproduced something weird with my importtest2.xls?

comment:28 Changed 11 months ago by jun66j5

I think that is a duplicate of #10194, not a ticketimportplugin issue.

Add Comment

Modify Ticket

Action
as reopened .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.