如何使用python将UTF-8文件分成单独的行(逗号分隔)？

Question

我正在尝试将UTF-16文件转换为UTF-8文件(因为我使用的是python csv模块，该模块显然不处理UTF-16文件)。然后，我想分隔这个UTF-8文件，以便可以使用一个简单的row.strip()方法将它导入到postgres表中。此python文件如下所示：

with codecs.open(sourcefile, 'rU', 'UTF-16') as infile:
   with open(sourcefile + '.utf8', 'wb') as outfile:
       for line in infile:
           outfile.write(line.encode('utf8'))


with open(sourcefile + '.utf8', 'rb') as f:
    reader = csv.reader(f, delimiter=',')

    for row in reader:
        print row[1]

我无法分隔行，因为行中似乎只有一个索引，而打印row1会给我一个索引范围-如何拆分这个文件？

Excel行：

15,"1/2 TYPE A","98","MCDS, TX","XA","852","TX","955","148","HAPPY, TX",,"$0.00","0","0.00","$1,504","179","0.00%","100.00%","0"
32,"1/2 TYPE B","98","MCDS, MI","XA","252","MI","72","925","HAPPY, MI",,"$0.00","0","0.00","$2,504","225","0.00%","100.00%","0"

很抱歉，我的描述不够详尽。基本上，输入文件是一个UTF-16文件。我曾经用excel打开文件，用分隔符'，‘将一列分成多列，然后将其保存为csv文件。然后，我通过能够读取csv文件、剥离行并将数据导入postgres数据库的python脚本运行经过处理的csv文件。

python脚本的原始导入部分(当我用分隔符'，‘分隔时)看起来像这样(简化版本)：

 for row in reader:
    arg = {
            'item_number': row[0].strip(),
            'item_size': row[1].strip(),
            'description': row[2].strip(),
            #etc...
        }
        cur.execute(
            """INSERT INTO 
            "Sales"("ITEM_NUMBER","ITEM_SIZE","DESCRIPTION"")
             select
                %(item_number)s, 
                %(item_size)s )
                %(description);""", arg)

但是，我现在希望能够简单地使用我的python脚本运行UTF-16文件，将数据导入postgres，这样我就完全不必在excel中打开该文件。我想这样做:将文件转换为UTF-8文件，然后以某种方式剥离每一行并将其导入到我的数据库中。

我已经能够成功地将该文件转换为UTF-8，但现在我遇到了一个问题，即UTF-8文件本质上是一堆被视为“一列”的行。我该如何剥离每一行呢？我不能做简单的row.strip()，因为在属于描述的文件中有一些逗号。

Answer 1

不需要创建中间文件，只需使用描述的转换in the docs (搜索unicode_csv_reader)。为方便起见，我将生成器转换为生成器表达式：

import codecs
import csv

sourcefile = 'csv16.csv'
with codecs.open(sourcefile, 'rU', 'UTF-16') as infile:
    reader = csv.reader((line.encode('utf-8')
                         for line in infile),
                        delimiter=',')
    for row in ([item.decode('utf-8')
                 for item in row]
                for row in reader):
        print u'/'.join(row)

我已经针对以下文件测试了上面的代码，该文件另存为Big-endian UTF-16：

1,2,3,4
5,6,7,8
"98°","①", "®©§™"

输出：

1/2/3/4
5/6/7/8
98°/①/ "®©§™"