Modify

Opened 15 years ago

Closed 8 years ago

#5617 closed defect (wontfix)

mantis2trac.py dies on Unicode errors

Reported by: tdmurphy4@… Owned by: Anton Stroganov
Priority: high Component: MantisImportScript
Severity: critical Keywords:
Cc: Trac Release: 0.12

Description (last modified by Ryan J Ollos)

I am trying to run mantis2trac.py on a Mantis v1.1.8 server running under Mysql 5.x. It imports 40 tickets, than on ticket 41 it bombs out with:

Traceback (most recent call last):
  File "./mantis2trac.py", line 943, in <module>
    main()
  File "./mantis2trac.py", line 940, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac.py", line 680, in convert
    trac.addTicket(**ticket)
  File "./mantis2trac.py", line 386, in addTicket
    desc = description.encode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa3 in position 244: unexpected code byte

I looked at ticket 41. It has the UK pound symbol (£) in it. How do I get the script to fix this properly? I'd like to import all of the tickets in but this is a really big showstopper.

Thanks!

Attachments (0)

Change History (5)

comment:1 Changed 15 years ago by tdmurphy4@…

I tried converting the Mantis database from latin1 to utf8 with the instructions given at http://en.gentoo-wiki.com/wiki/TIP_Convert_latin1_to_UTF-8_in_MySQL but it makes no difference. The python script completely barfs when it hits a '£'. Is there a way around this?

comment:2 Changed 15 years ago by tdmurphy4@…

I managed to fix it. But you might want to add notes to the mantis2trac.py page that if Mysql has the database in latin1 format, and Mantis merrily encodes things with UTF8 anyway, that dumping and adding the database per instructions here: http://paulkortman.com/2009/07/24/mysql-latin1-to-utf8-conversion/ fixes things.

I basically dumped with: --default-character-set=utf8 then ran sed on the dump to change occurrances of utf8 to latin1 (sed -i 's/utf8/latin1/g' dump.sql), then imported it into a new, empty database I created and ran the mantis import script on that.

comment:3 Changed 15 years ago by anonymous

Trac Release: 0.110.12

I too experience this problem, with the Mantis database's table all encoded using the utf8_general_ci encoding. I've tried to just remove the calls to .encode('utf-8'), but this results in another error altogether:

inserting ticket 91 -- "speciale teken &#966; als ? afgedrukt"
Traceback (most recent call last):
  File "c:\python26\scripts\mantis2trac.py", line 946, in <module>
    main()
  File "c:\python26\scripts\mantis2trac.py", line 943, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "c:\python26\scripts\mantis2trac.py", line 683, in convert
    trac.addTicket(**ticket)
  File "c:\python26\scripts\mantis2trac.py", line 404, in addTicket
    summary, desc, keywords))
  File "build\bdist.win32\egg\trac\db\util.py", line 122, in execute
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (l
ike text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

This occurs on special characters like ë and á and ì. Is there any quick fix to be done in the script to fix this? I'm no character encoding expert, and I've already spent far too much time on this problem.

Last edited 8 years ago by Ryan J Ollos (previous) (diff)

comment:4 Changed 8 years ago by Ryan J Ollos

Description: modified (diff)

comment:5 Changed 8 years ago by Ryan J Ollos

Resolution: wontfix
Status: newclosed

New version of plugin is on GitHub. Please open a new ticket if you have an issue with the new version.

Modify Ticket

Change Properties
Set your email in Preferences
Action
as closed The owner will remain Anton Stroganov.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.