Ticket #10188 (reopened defect)

Opened 10 months ago

Last modified 2 months ago

simple import failed

Reported by: falkb Assigned to: farialima
Priority: normal Component: TicketImportPlugin
Severity: major Keywords:
Cc: jun66j5 Trac Release: 0.12

Description

The file is the attached .xls one. The callstack here is: 

Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\api.py", line 440, in send_error
    data, 'text/html')
  File "build\bdist.win32\egg\trac\web\chrome.py", line 868, in render_template
    stream.render(method, doctype=doctype, out=buffer)
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 183, in render
    return encode(generator, method=method, encoding=encoding, out=out)
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 58, in encode
    for chunk in iterator:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 339, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 826, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 670, in __call__
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 771, in __call__
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "build/bdist.macosx-10.6-i386/egg/genshi/output.py", line 586, in __call__
    for ev in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 981, in _strip_accesskeys
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build\bdist.win32\egg\trac\web\chrome.py", line 970, in _generate
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 378, in _match
    ctxt, start=idx + 1, **vars):
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 327, in _match
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 545, in _flatten
    for kind, data, pos in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/path.py", line 588, in _generate
    subevent = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 605, in _include
    for event in stream:
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/markup.py", line 316, in _strip
    event = next()
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 565, in _flatten
    result = _eval_expr(data, ctxt, vars)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/base.py", line 277, in _eval_expr
    retval = expr.evaluate(ctxt)
  File "build/bdist.macosx-10.6-i386/egg/genshi/template/eval.py", line 178, in evaluate
    return eval(self.code, _globals, {'__data__': data})
  File "d:/trac/test_devel\.egg-cache\Trac-0.12.2-py2.5.egg-tmp\trac\templates\error.html", line 165, in <Expression u'shorten_line(repr(value))'>
    <td><code>${shorten_line(repr(value))}</code></td>
  File "build\bdist.win32\egg\mastertickets\model.py", line 118, in __repr__
    (self.tkt.id, l(getattr(self, 'blocking', [])), l(getattr(self, 'blocked_by', [])))
  File "build\bdist.win32\egg\mastertickets\model.py", line 115, in l
    return '[%s]'%','.join(arr2)
TypeError: sequence item 0: expected string, int found

It's Trac-0.12.2 on Windows, TicketImportPlugin is the latest from SVN, TracMasterTickets?-3.0.2-py2.5.egg is the latest available.

Attachments

bug.xls (29.0 kB) - added by falkb on 07/26/12 11:28:58.
to-unicode-if-custom-field-r11799.diff (1.3 kB) - added by jun66j5 on 07/26/12 13:28:50.
importtest2.xls (26.0 kB) - added by falkb on 04/08/13 10:56:19.
more complex file waking up an integrity bug

Change History

07/26/12 11:28:58 changed by falkb

  • attachment bug.xls added.

07/26/12 13:28:50 changed by jun66j5

  • attachment to-unicode-if-custom-field-r11799.diff added.

07/26/12 13:31:44 changed by jun66j5

Thanks for the report. If the value of custom fields is not unicode, the issue will occur.

Could you please try the patch, to-unicode-if-custom-field-r11799.diff? (I will add unit tests for it later.)

(follow-up: ↓ 3 ) 07/26/12 13:45:04 changed by falkb

No, sorry, the patch didn't help. The callstack remains the same.

(in reply to: ↑ 2 ) 07/26/12 13:57:13 changed by jun66j5

Sorry. I've misunderstood.

It seems that the issue has been fixed in https://github.com/coderanger/trac-mastertickets/commit/634d8784aed8f5a77cb3ffc470241bdab253c829. Could you please try the latest source from https://github.com/coderanger/trac-mastertickets/zipball/master?

07/26/12 14:06:35 changed by falkb

Previously, I used masterticket version coderanger-trac-mastertickets-42b59b4 . Now I tried your recommended coderanger-trac-mastertickets-34803ad . Your patch of 1 is still active.

The error on importing bug.xls turns to: IntegrityError: columns ticket, time, field are not unique

with callstack: 

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 138, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

07/26/12 16:07:09 changed by farialima

I think I know what the problem is... kind of my fault, I modified the patch when applying - I set the modification date when adding the master ticket info to the ticket, to be the same as the modification date of the first import... give me a few hours to fix it, I'm quite busy currently, but I'll get to it today.

07/27/12 12:28:34 changed by falkb

Any news?

07/27/12 19:19:14 changed by farialima

  • status changed from new to closed.
  • resolution set to fixed.

ok I've implemented a workaround in [11805]. In fact, it's not my fault, it's a bug in MasterTicketPlugin?, and I have to work around it... I'll enter it...

07/27/12 19:25:29 changed by farialima

Root issue entered as #10194

07/27/12 21:36:30 changed by falkb

Thanks. Unfortunately, this working week is over, and I wont be able to test it for some time now...

09/10/12 10:56:42 changed by falkb

  • status changed from closed to reopened.
  • resolution deleted.

Now I was able to work further on this problem. The latest SVN version still doesn't work for that attached bug.xls

There's still 

IntegrityError: columns ticket, time, field are not unique

The callstack is slightly different to the one of 4 

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 73, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

(follow-up: ↓ 12 ) 09/12/12 13:13:28 changed by farialima

I've trried to reproduce your problem, but to no avail

What version of Trac are you running ? what version of the MasterTickets? plugin ?

thanks

(in reply to: ↑ 11 ) 09/12/12 13:18:12 changed by falkb

Replying to farialima:

What version of Trac are you running ?

Bitnami 0.12.2, Apache, Windows

what version of the MasterTickets? plugin ?

coderanger-trac-mastertickets-34803ad

(follow-up: ↓ 14 ) 09/12/12 13:36:16 changed by falkb

Well, unfortunately, there is no SVN version of the mastertickets plugin that can be checked for local modifications... so I compared again a newly downloaded 'coderanger-trac-mastertickets-34803ad' from web against my local copy of 'coderanger-trac-mastertickets-34803ad' and found out somewhen ago I obviously applied a small patch in def validate_ticket in api.py:135. Maybe that's the reason?: 

        # Check that ticket does not have itself as a blocker 
-        if id in links.blocking | links.blocked_by:
-            yield 'blocked_by', 'This ticket is blocking itself'
+        if id.isdecimal() and int(id) in links.blocking:
+            yield 'blocking', 'This ticket is blocking itself'
+            return
+            
+        if id.isdecimal() and int(id) in links.blocked_by:
+            yield 'blocked_by', 'This ticket is blocked by itself'
            return

I'll retry bug.xls

(in reply to: ↑ 13 ; follow-up: ↓ 15 ) 09/12/12 13:53:34 changed by falkb

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch): 

Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 511, in _dispatch_request
  dispatcher.dispatch(req)
Datei "build/bdist.win32/egg/trac/web/main.py", Zeile 237, in dispatch
  resp = chosen_handler.process_request(req)
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 86, in process_request
  encoding=req.session['importer.encoding'])
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
Datei "build/bdist.win32/egg/talm_importer/importer.py", Zeile 401, in _process
  processor.end_process_row()
Datei "build/bdist.win32/egg/talm_importer/processors.py", Zeile 133, in end_process_row
  self.ticket.insert(when=self._tickettime(), db=self.db)
Datei "build/bdist.win32/egg/trac/ticket/model.py", Zeile 246, in insert
  listener.ticket_created(self)
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 106, in ticket_created
  self.ticket_changed(tkt, '', tkt['reporter'], {})
Datei "build/bdist.win32/egg/mastertickets/api.py", Zeile 111, in ticket_changed
  links.save(author, comment, tkt.time_changed, db)
Datei "build/bdist.win32/egg/mastertickets/model.py", Zeile 72, in save
  (n, when_ts, author, field, old_value, new_value))
Datei "build/bdist.win32/egg/trac/db/util.py", Zeile 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 78, in execute
  result = PyFormatCursor.execute(self, *args)
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 56, in execute
  args or [])
Datei "build/bdist.win32/egg/trac/db/sqlite_backend.py", Zeile 48, in _rollback_on_error
  return function(self, *args, **kwargs)

Systeminformationen:

User Agent: Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1 

Trac 	0.12.2
Babel 	0.9.5
Bitten 	0.7dev-r1008
FullBlog 	0.1.1-r9430
Genshi 	0.6
mod_python 	3.3.1
pysqlite 	2.5.5
Python 	2.5.4 (r254:67916, Dec 23 2008, 15:10:54) [MSC v.1310 32 bit (Intel)]
RPC 	1.1.0
setuptools 	0.6c11
SQLite 	3.6.11
Subversion 	1.6.6 (r40053)
jQuery	1.4.2

(in reply to: ↑ 14 ) 09/12/12 14:03:09 changed by falkb

Replying to falkb:

Replying to falkb:

I'll retry bug.xls

No, nearly the same result using latest SVN version r11981 of ticketimportplugin (0.11 branch)

and the original version of coderanger-trac-mastertickets-34803ad

10/29/12 13:49:00 changed by falkb

farialima, do you still have an idea how to get further here?

10/29/12 15:45:30 changed by rjollos

I'm maintaining the MasterTicketsPlugin since very recently, and I acknowledge the defect reported in #10194. I've been able to reproduce it several times and hope to have the issue fixed in the next version.

03/13/13 13:43:27 changed by falkb

rjollos, once you said you have got the fix almost ready at your machine and just need to commit then. Is there hope to get it somehow these days? To fix this issue here is quite important...

03/13/13 14:17:51 changed by rjollos

Unfortunately I lost that, and quite a bit of other work that I had in local Git repositories, when my virtual machine stopped booting. I intend to recreate the patches, and it is fairly high on my priority list, but I can't say for sure when it will be ready.

03/22/13 02:16:28 changed by rjollos

Should be fixed in MasterTicketsPlugin 3.0.4. Please report back if you have a chance to test.

03/27/13 09:29:42 changed by falkb

Still has problems as reported in comment:ticket:10194:10. I used attachment:bug.xls for my test but it fails with that reported callstack.

04/04/13 10:06:07 changed by rjollos

This might be fixed in [12950]. I'm not very happy with the solution, but please give it a try and let me know if it is working for you now.

(follow-up: ↓ 24 ) 04/08/13 09:02:51 changed by falkb

Now using todays latest SVN version of TH's MasterTicketsPlugin (r12950) and TicketImportPlugin (r12819) I tested attachment:bug.xls and the import of it works fine for it now => Thanks a lot!

Though trying a more complex .xls file I run into this integrity error: 

File "build/bdist.win32/egg/trac/web/main.py", line 497, in _dispatch_request
  dispatcher.dispatch(req)
File "build/bdist.win32/egg/trac/web/main.py", line 214, in dispatch
  resp = chosen_handler.process_request(req)
File "build/bdist.win32/egg/talm_importer/importer.py", line 86, in process_request
  encoding=req.session['importer.encoding'])
File "build/bdist.win32/egg/talm_importer/importer.py", line 130, in _do_import
  return self._process(filereader, get_reporter_id(req), ImportProcessor(self.env, req, uploadedfilename, tickettime))
File "build/bdist.win32/egg/talm_importer/importer.py", line 417, in _process
  processor.process_relativeticket_fields(relativeticketvalues, lowercaserelativeticketfields)
File "build/bdist.win32/egg/talm_importer/processors.py", line 237, in process_relativeticket_fields
  self._save_ticket(ticket, with_comment=False)
File "build/bdist.win32/egg/talm_importer/processors.py", line 113, in _save_ticket
  db=self.db)
File "build/bdist.win32/egg/trac/ticket/model.py", line 351, in save_changes
  self[name]))
File "build/bdist.win32/egg/trac/db/util.py", line 121, in execute
  cursor.execute(query, params)
File "build/bdist.win32/egg/trac/db/util.py", line 65, in execute
  return self.cursor.execute(sql_escape_percent(sql), args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 78, in execute
  result = PyFormatCursor.execute(self, *args)
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 56, in execute
  args or [])
File "build/bdist.win32/egg/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
  return function(self, *args, **kwargs)

(in reply to: ↑ 23 ; follow-up: ↓ 25 ) 04/08/13 10:05:02 changed by rjollos

Replying to falkb:

... Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

04/08/13 10:56:19 changed by falkb

  • attachment importtest2.xls added.

more complex file waking up an integrity bug

(in reply to: ↑ 24 ; follow-up: ↓ 26 ) 04/08/13 11:02:09 changed by falkb

Replying to rjollos:

Replying to falkb:

... Though trying a more complex .xls file I run into this integrity error:

Could you attach that file?

I had to decorate the original cell content a bit (confidential data), uploaded it as attachment:importtest2.xls, but I'm still able to run into the same callstack with that new file. Although a bit of the import work seems to be done because the timeline says: 

 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb
    blocking changed

(in reply to: ↑ 25 ) 04/08/13 11:15:28 changed by falkb

Replying to falkb:

... the timeline says: {{{ 10:50 AM Tickets #2747,​2910,​4265,​4266,​4272 batch updated by falkb blocking changed }}}

I wonder why the timeline does not report the creation of 4265,​4266 and ​4272. The latest ticket before the import was 4263.

Add/Change #10188 (simple import failed)




Change Properties
Action