Ticket #4458 (closed defect: fixed)

Opened 3 years ago

Last modified 1 year ago

UnicodeDecodeError

Reported by: weltweitehrlich Assigned to: farialima
Priority: normal Component: TicketImportPlugin
Severity: normal Keywords:
Cc: Trac Release: 0.11

Description

We have german umlauts. Importing a exported unmodified CSV generates: 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)}}}

Attachments

query.csv (122 bytes) - added by weltweitehrlich on 01/27/09 04:52:35.
Example file for this error

Change History

(follow-ups: ↓ 2 ↓ 4 ) 01/15/09 12:21:52 changed by farialima

CSV files and non-ascii codecs are tricky. I thought that it was working, but it seems that it's not perfect.

Could you attach a CSV file that exposes the problem ? Also, are you on Python 2.4, 2.5 or 2.6 ?

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

01/27/09 04:52:35 changed by weltweitehrlich

  • attachment query.csv added.

Example file for this error

(in reply to: ↑ 1 ) 01/27/09 04:58:41 changed by weltweitehrlich

Replying to farialima:

Could you attach a CSV file that exposes the problem ?

Done.

Also, are you on Python 2.4, 2.5 or 2.6 ?

I'm using python 2.5

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

I have no xlrd installation at the moment. I will test it later.

(follow-up: ↓ 6 ) 02/12/09 08:56:21 changed by giangnh@aureole-net.com.vn

I met exactly same problem in Japanese(even use xlrd).

(in reply to: ↑ 1 ; follow-up: ↓ 5 ) 02/23/09 09:14:34 changed by weltweitehrlich

Replying to farialima:

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

The Excel spreadsheet seems to work. You are right. The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

(in reply to: ↑ 4 ; follow-up: ↓ 7 ) 02/23/09 20:29:47 changed by farialima

Replying to weltweitehrlich:

The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

I've committed a fix and updated the ".egg" files on the Wiki page. Please try it and tell me if it works for you.

(in reply to: ↑ 3 ) 02/23/09 20:31:33 changed by farialima

Replying to giangnh@aureole-net.com.vn:

I met exactly same problem in Japanese(even use xlrd).

Could you attach a .csv file, and an .xls file, to this ticket, so that I can look at the problem ?

I don't think that the fix I just did will work for you: my fix supposes that the .csv file uses a European encoding. However, if you give me some sample files, I will try to fix it for Japanese also.

(in reply to: ↑ 5 ) 02/26/09 10:29:50 changed by weltweitehrlich

Replying to farialima:

Replying to weltweitehrlich:

The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

I've committed a fix and updated the ".egg" files on the Wiki page. Please try it and tell me if it works for you.

The error is gone. That's the good point. The bad one is that the encoding seems to be wrong. Maybe my fault. I must check this.

04/23/10 20:20:09 changed by farialima

Any news on this one ? can I close it as fixed ?

(follow-up: ↓ 11 ) 09/15/10 19:27:47 changed by anonymous

tried import of a Japanese file. csv format doesn't work, but Excel 2003 works fine with xlrd.

10/18/10 20:52:31 changed by ruidpires@gmail.com

Problem still happening if the file has the character u\xea which translates to 'ê'

Other than that, it seems to be working.

(in reply to: ↑ 9 ) 01/04/11 03:35:29 changed by farialima

  • status changed from new to assigned.

Based on anonymous:

tried import of a Japanese file. csv format doesn't work, but Excel 2003 works fine with xlrd.

In #4458 I've added a warning:

"(Excel is preferred if your data contain non-ASCII characters). "

because there will be *always* issues with charsets in CSV - it does not support full Unicode.

I think we can mark this as fixed... anyone disagreeing can reopen it :)

01/04/11 03:35:36 changed by farialima

  • status changed from assigned to closed.
  • resolution set to fixed.

Add/Change #4458 (UnicodeDecodeError)




Change Properties
Action