Opened 15 years ago
Closed 8 years ago
#5617 closed defect (wontfix)
mantis2trac.py dies on Unicode errors
Reported by: | Owned by: | Anton Stroganov | |
---|---|---|---|
Priority: | high | Component: | MantisImportScript |
Severity: | critical | Keywords: | |
Cc: | Trac Release: | 0.12 |
Description (last modified by )
I am trying to run mantis2trac.py on a Mantis v1.1.8 server running under Mysql 5.x. It imports 40 tickets, than on ticket 41 it bombs out with:
Traceback (most recent call last): File "./mantis2trac.py", line 943, in <module> main() File "./mantis2trac.py", line 940, in main convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN) File "./mantis2trac.py", line 680, in convert trac.addTicket(**ticket) File "./mantis2trac.py", line 386, in addTicket desc = description.encode('utf-8') UnicodeDecodeError: 'utf8' codec can't decode byte 0xa3 in position 244: unexpected code byte
I looked at ticket 41. It has the UK pound symbol (£) in it. How do I get the script to fix this properly? I'd like to import all of the tickets in but this is a really big showstopper.
Thanks!
Attachments (0)
Change History (5)
comment:1 Changed 15 years ago by
comment:2 Changed 15 years ago by
I managed to fix it. But you might want to add notes to the mantis2trac.py page that if Mysql has the database in latin1 format, and Mantis merrily encodes things with UTF8 anyway, that dumping and adding the database per instructions here: http://paulkortman.com/2009/07/24/mysql-latin1-to-utf8-conversion/ fixes things.
I basically dumped with: --default-character-set=utf8 then ran sed on the dump to change occurrances of utf8 to latin1 (sed -i 's/utf8/latin1/g' dump.sql), then imported it into a new, empty database I created and ran the mantis import script on that.
comment:3 Changed 15 years ago by
Trac Release: | 0.11 → 0.12 |
---|
I too experience this problem, with the Mantis database's table all encoded using the utf8_general_ci encoding. I've tried to just remove the calls to .encode('utf-8'), but this results in another error altogether:
inserting ticket 91 -- "speciale teken φ als ? afgedrukt" Traceback (most recent call last): File "c:\python26\scripts\mantis2trac.py", line 946, in <module> main() File "c:\python26\scripts\mantis2trac.py", line 943, in main convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN) File "c:\python26\scripts\mantis2trac.py", line 683, in convert trac.addTicket(**ticket) File "c:\python26\scripts\mantis2trac.py", line 404, in addTicket summary, desc, keywords)) File "build\bdist.win32\egg\trac\db\util.py", line 122, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (l ike text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
This occurs on special characters like ë and á and ì. Is there any quick fix to be done in the script to fix this? I'm no character encoding expert, and I've already spent far too much time on this problem.
comment:4 Changed 8 years ago by
Description: | modified (diff) |
---|
comment:5 Changed 8 years ago by
Resolution: | → wontfix |
---|---|
Status: | new → closed |
New version of plugin is on GitHub. Please open a new ticket if you have an issue with the new version.
I tried converting the Mantis database from latin1 to utf8 with the instructions given at http://en.gentoo-wiki.com/wiki/TIP_Convert_latin1_to_UTF-8_in_MySQL but it makes no difference. The python script completely barfs when it hits a '£'. Is there a way around this?