使用.csv读取星火中的sqlContext.read文件时出错

Question

我正试图将csv文件读入星火中的数据文件，如下所示：

1. 我运行火花壳，就像： 星星之弹-- .\spark-csv_2.11-1.4.0.jar;.\commons-csv-1.2.jar (我不能直接下载那些依赖项，这就是我使用--jars的原因)
2. 使用以下命令读取csv文件：

( sqlContext.read.format("com.databricks.spark.csv").option("header"，"true").load("2008.csv")

但是，下面是我得到的错误消息：

scala> val df_1 = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load("2008.csv")
java.lang.NoClassDefFoundError: org/apache/commons/csv/CSVFormat
        at com.databricks.spark.csv.package$.<init>(package.scala:27)
        at com.databricks.spark.csv.package$.<clinit>(package.scala)
        at com.databricks.spark.csv.CsvRelation.inferSchema(CsvRelation.scala:235)
        at com.databricks.spark.csv.CsvRelation.<init>(CsvRelation.scala:73)
        at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:162)
        at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:44)
        at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
        at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:41)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:43)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:45)
        at $iwC$$iwC$$iwC.<init>(<console>:47)
        at $iwC$$iwC.<init>(<console>:49)
        at $iwC.<init>(<console>:51)
        at <init>(<console>:53)
        at .<init>(<console>:57)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop
.scala:997)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:
945)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:
945)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.csv.CSVFormat
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 57 more

在完成第一个提议的解决方案之后：

PS C:\Users\319413696\Desktop\graphX> spark-shell --packages com.databricks:spark-csv_2.11:1.4.0
Ivy Default Cache set to: C:\Users\319413696\.ivy2\cache
The jars for the packages stored in: C:\Users\319413696\.ivy2\jars
:: loading settings :: url = jar:file:/C:/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar!/org/apache
/ivy/core/settings/ivysettings.xml
com.databricks#spark-csv_2.11 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
        confs: [default]
        found com.databricks#spark-csv_2.11;1.4.0 in local-m2-cache
        found org.apache.commons#commons-csv;1.1 in local-m2-cache
        found com.univocity#univocity-parsers;1.5.1 in local-m2-cache
downloading file:/C:/Users/319413696/.m2/repository/com/databricks/spark-csv_2.11/1.4.0/spark-csv_2.11-1.4.0.jar ...
        [SUCCESSFUL ] com.databricks#spark-csv_2.11;1.4.0!spark-csv_2.11.jar (0ms)
downloading file:/C:/Users/319413696/.m2/repository/org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar ...
        [SUCCESSFUL ] org.apache.commons#commons-csv;1.1!commons-csv.jar (0ms)
downloading file:/C:/Users/319413696/.m2/repository/com/univocity/univocity-parsers/1.5.1/univocity-parsers-1.5.1.jar ..
.
        [SUCCESSFUL ] com.univocity#univocity-parsers;1.5.1!univocity-parsers.jar (15ms)
:: resolution report :: resolve 671ms :: artifacts dl 31ms
        :: modules in use:
        com.databricks#spark-csv_2.11;1.4.0 from local-m2-cache in [default]
        com.univocity#univocity-parsers;1.5.1 from local-m2-cache in [default]
        org.apache.commons#commons-csv;1.1 from local-m2-cache in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   3   |   3   |   3   |   0   ||   3   |   3   |
        ---------------------------------------------------------------------

:: problems summary ::
:::: ERRORS
        Server access error at url https://repo1.maven.org/maven2/com/databricks/spark-csv_2.11/1.4.0/spark-csv_2.11-1.4
.0-sources.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/databricks/spark-csv_2.11/1.4.0/spark-
csv_2.11-1.4.0-sources.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/com/databricks/spark-csv_2.11/1.4.0/spark-csv_2.11-1.4
.0-src.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/databricks/spark-csv_2.11/1.4.0/spark-
csv_2.11-1.4.0-src.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/com/databricks/spark-csv_2.11/1.4.0/spark-csv_2.11-1.4
.0-javadoc.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/databricks/spark-csv_2.11/1.4.0/spark-
csv_2.11-1.4.0-javadoc.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/org/apache/apache/15/apache-15.jar (java.net.SocketExc
eption: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/apache/15/apache-15.jar (java.n
et.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35
.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/commons/commons-parent/35/commo
ns-parent-35.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/org/apache/commons/commons-csv/1.1/commons-csv-1.1-sou
rces.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/commons/commons-csv/1.1/commons
-csv-1.1-sources.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/org/apache/commons/commons-csv/1.1/commons-csv-1.1-src
.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/commons/commons-csv/1.1/commons
-csv-1.1-src.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/org/apache/commons/commons-csv/1.1/commons-csv-1.1-jav
adoc.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/org/apache/commons/commons-csv/1.1/commons
-csv-1.1-javadoc.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/com/univocity/univocity-parsers/1.5.1/univocity-parser
s-1.5.1-sources.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/univocity/univocity-parsers/1.5.1/univ
ocity-parsers-1.5.1-sources.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/com/univocity/univocity-parsers/1.5.1/univocity-parser
s-1.5.1-src.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/univocity/univocity-parsers/1.5.1/univ
ocity-parsers-1.5.1-src.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url https://repo1.maven.org/maven2/com/univocity/univocity-parsers/1.5.1/univocity-parser
s-1.5.1-javadoc.jar (java.net.SocketException: Permission denied: connect)

        Server access error at url http://dl.bintray.com/spark-packages/maven/com/univocity/univocity-parsers/1.5.1/univ
ocity-parsers-1.5.1-javadoc.jar (java.net.SocketException: Permission denied: connect)

Answer 1

我没有使用sqlContext.read，而是使用以下代码将.csv文件转换为数据格式。假设.csv文件有5列，如下所示：

case class Flight(arrDelay: Int, depDelay: Int, origin: String, dest: String, distance: Int)

然后：

val flights=sc.textFile("2008.csv").map(_.split(",")).map(p => Flight(p(0).trim.toInt, p(1).trim.toInt, p(2)
, p(3), p(4).trim.toInt)).toDF()

Answer 2

1. 给罐子的全部道路，并将他们分开，而不是； 火花-壳-罐火花-壳-罐满路径\火花-csv_2.11-1.4.0.jar，满路径\共用-csv-1.2.jar
2. 确保您在文件夹(DFS)中具有写入tempory文件的权限。

Answer 3

下载星星之火-csv到您的.m2目录，然后使用spark-shell --packages com.databricks:spark-csv_2.11:1.4.0

如果你不能直接下载火花-csv，下载它在其他系统和复制所有.m2目录到您的计算机。