读取包含双引号但带有换行符的csv文件

Question

我有csv，列中有换行符。下面是我的示例：

"A","B","C"
1,"This is csv with 
newline","This is another column"
"This is newline
and another line","apple","cat"

我可以在spark中读取文件，但是列中的换行符被看作是一个单独的行。

我如何准备作为一个csv与文字封闭在双引号内。

我只使用apache插件和apache读取文件。

alarms = sc.textFile("D:\Dataset\oneday\oneday.csv")

这给了我RDD：

**example.take(5)**

[u'A,B,C', u'1,"This is csv with ', u'newline",This is another column', u'"This is newline', u'and another line",apple,cat']

火花版本: 1.4

Answer 1

如果您想用新行从csv创建数据，并以双引号引用，而不需要重新创建车轮，那么请使用spark和公共-csv库：

from pyspark.sql import SQLContext
df = sqlContext.load(header="true",source="com.databricks.spark.csv", path = "hdfs://analytics.com.np:8020/hdp/badcsv.csv")

Answer 2

标准python库中的csv模块可以不加限制地完成它：

>>> txt = '''"A","B","C"
1,"This is csv with 
newline","This is another column"
"This is newline
and another line","apple","cat"'''
>>> import csv
>>> import io
>>> with io.BytesIO(txt) as fd:
    rd = csv.reader(fd)
    for row in rd:
        print row


['A', 'B', 'C']
['1', 'This is csv with \nnewline', 'This is another column']
['This is newline\nand another line', 'apple', 'cat']

这可以与binaryFiles一起使用(与binaryFiles相比，性能损失很大)。

>>> (sc.binaryFiles(path)
        .values()
        .flatMap(lambda x: csv.reader(io.BytesIO(x))))

Answer 3

您不需要导入任何东西。下面提出的解决方案仅为演示目的创建了第二个文件。您可以在修改后读取该行，而无需将其写入任何地方。

with open(r'C:\Users\evkouni\Desktop\test_in.csv', 'r') as fin:
    with open(r'C:\Users\evkouni\Desktop\test_out.csv', 'w') as fout:
        cont = fin.readlines()
        for line in cont[:-1]:
            if line.count('"') % 2 == 1 and '"\n' not in line:
                line = line.replace('\n', '')
            fout.write(line)

#DEMO

#test_in.csv
#------------
#"A";"B";"C"
#1;"This is csv with 
#newline";"This is another column"
#"This is newline

#test_out.csv
#------------
#"A";"B";"C"
#1;"This is csv with newline";"This is another column"
#"This is newline

如果你有什么不清楚的地方，请告诉我。