UnicodeDecodeError

[weltweitehrlich]

We have german umlauts. Importing a exported unmodified CSV generates:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 11: ordinal not in range(128)}}}

[François Granade]

CSV files and non-ascii codecs are tricky. I thought that it was working, but it seems that it's not perfect.

Could you attach a CSV file that exposes the problem ? Also, are you on Python 2.4, 2.5 or 2.6 ?

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

[weltweitehrlich]

Example file for this error

[weltweitehrlich]

Replying to farialima:

Could you attach a CSV file that exposes the problem ?

Done.

Also, are you on Python 2.4, 2.5 or 2.6 ?

I'm using python 2.5

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

I have no xlrd installation at the moment. I will test it later.

[giangnh@…]

I met exactly same problem in Japanese(even use xlrd).

[weltweitehrlich]

Replying to farialima:

I think that an Excel spreadsheet would work - that's probably a workaround for you ?

The Excel spreadsheet seems to work. You are right. The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

[François Granade]

Replying to weltweitehrlich:

The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

I've committed a fix and updated the ".egg" files on the Wiki page. Please try it and tell me if it works for you.

[François Granade]

Replying to giangnh@aureole-net.com.vn:

I met exactly same problem in Japanese(even use xlrd).

Could you attach a .csv file, and an .xls file, to this ticket, so that I can look at the problem ?

I don't think that the fix I just did will work for you: my fix supposes that the .csv file uses a European encoding. However, if you give me some sample files, I will try to fix it for Japanese also.

[weltweitehrlich]

Replying to farialima:

Replying to weltweitehrlich:

The ticket export is CSV so it's not the best process to import using Excel. Can you fix it?

I've committed a fix and updated the ".egg" files on the Wiki page. Please try it and tell me if it works for you.

The error is gone. That's the good point. The bad one is that the encoding seems to be wrong. Maybe my fault. I must check this.

[François Granade]

Any news on this one ? can I close it as fixed ?

[anonymous]

tried import of a Japanese file. csv format doesn't work, but Excel 2003 works fine with xlrd.

[ruidpires@…]

Problem still happening if the file has the character u\xea which translates to 'ê'

Other than that, it seems to be working.

[François Granade]

Based on anonymous:

tried import of a Japanese file. csv format doesn't work, but Excel 2003 works fine with xlrd.

In #4458 I've added a warning:

"(Excel is preferred if your data contain non-ASCII characters). "

because there will be *always* issues with charsets in CSV - it does not support full Unicode.

I think we can mark this as fixed... anyone disagreeing can reopen it :)