Modify

Opened 5 years ago

Last modified 4 years ago

#5617 new defect

mantis2trac.py dies on Unicode errors

Reported by: tdmurphy4@… Owned by: codeFiend
Priority: high Component: MantisImportScript
Severity: critical Keywords:
Cc: Trac Release: 0.12

Description

I am trying to run mantis2trac.py on a Mantis v1.1.8 server running under Mysql 5.x. It imports 40 tickets, than on ticket 41 it bombs out with:

Traceback (most recent call last):

File "./mantis2trac.py", line 943, in <module>

main()

File "./mantis2trac.py", line 940, in main

convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)

File "./mantis2trac.py", line 680, in convert

trac.addTicket(ticket)

File "./mantis2trac.py", line 386, in addTicket

desc = description.encode('utf-8')

UnicodeDecodeError: 'utf8' codec can't decode byte 0xa3 in position 244: unexpected code byte

I looked at ticket 41. It has the UK pound symbol (£) in it. How do I get the script to fix this properly? I'd like to import all of the tickets in but this is a really big showstopper.

Thanks!

Attachments (0)

Change History (3)

comment:1 Changed 5 years ago by tdmurphy4@…

I tried converting the Mantis database from latin1 to utf8 with the instructions given at http://en.gentoo-wiki.com/wiki/TIP_Convert_latin1_to_UTF-8_in_MySQL but it makes no difference. The python script completely barfs when it hits a '£'. Is there a way around this?

comment:2 Changed 5 years ago by tdmurphy4@…

I managed to fix it. But you might want to add notes to the mantis2trac.py page that if Mysql has the database in latin1 format, and Mantis merrily encodes things with UTF8 anyway, that dumping and adding the database per instructions here: http://paulkortman.com/2009/07/24/mysql-latin1-to-utf8-conversion/ fixes things.

I basically dumped with: --default-character-set=utf8 then ran sed on the dump to change occurrances of utf8 to latin1 (sed -i 's/utf8/latin1/g' dump.sql), then imported it into a new, empty database I created and ran the mantis import script on that.

comment:3 Changed 4 years ago by anonymous

  • Trac Release changed from 0.11 to 0.12

I too experience this problem, with the Mantis database's table all encoded using the utf8_general_ci encoding. I've tried to just remove the calls to .encode('utf-8'), but this results in another error altogether:

inserting ticket 91 -- "speciale teken &#966; als ? afgedrukt" Traceback (most recent call last):

File "c:\python26\scripts\mantis2trac.py", line 946, in <module>

main()

File "c:\python26\scripts\mantis2trac.py", line 943, in main

convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)

File "c:\python26\scripts\mantis2trac.py", line 683, in convert

trac.addTicket(ticket)

File "c:\python26\scripts\mantis2trac.py", line 404, in addTicket

summary, desc, keywords))

File "build\bdist.win32\egg\trac\db\util.py", line 122, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error

sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (l ike text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

This occurs on special characters like ë and á and ì. Is there any quick fix to be done in the script to fix this? I'm no character encoding expert, and I've already spent far too much time on this problem.

Add Comment

Modify Ticket

Action
as new The owner will remain codeFiend.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.