为什么火花提交失败与“未能加载类的数据源: org.apache.spark.sql.cassandra”与卡桑德拉连接器在-jars？

Question

火花版本: 1.4.1

卡桑德拉版本: 2.1.8

Datastax Cassandra连接器: 1.4.2-SNAPSHOT.jar

我跑的命令

./spark提交--jars /usr/local/src/spark-cassandra-connector/spark-cassandra-connector-java/target/scala-2.10/spark-cassandra-connector-java-assembly-1.4.2-SNAPSHOT.jar --驱动程序--类路径/usr/local/src/spark-cassandra-connector/spark-cassandra-connector-java/target/scala-2.10/spark-cassandra-connector-java-assembly-1.4.2-SNAPSHOT.jar --jars /usr/local/lib/spark-1.4.1/external/kafka/target/scala-2.10/spark-streaming-kafka_2.10-1.4.1.jar --jars /usr/local/lib/spark-1.4.1/external/kafka-assembly/target/scala-2.10/spark-streaming-kafka-assembly_2.10-1.4.1.jar --驱动程序类路径/usr/local/lib/spark-1.4.1/external/kafka/target/scala-2.10/火花流-kafka_2.10-1.4.1.jar-驱动-类-路径/usr/local/lib/spark-1.4.1/external/kafka-assembly/target/scala-2.10/spark-streaming-kafka-assembly_2.10-1.4.1.jar -包org.apache.spark:spark-streaming-kafka_2.10:1.4.1 -执行程序-内存6g -执行器-核心6-主local4 kafka_streaming.py

下面是我遇到的错误：

Py4JJavaError: An error occurred while calling o169.save.
: java.lang.RuntimeException: Failed to load class for data source: org.apache.spark.sql.cassandra

一定是在做傻事。如有任何答复，将不胜感激。

Answer 1

尝试在相同的

--jars yourFirstJar.jar,yourSecondJar.jar

用于开发目的的一个更方便的解决方案是使用来自maven central (逗号分隔的)的jars：

--packages org.apache.spark:spark-streaming-kafka_2.10:1.4.1,com.datastax.spark:spark-cassandra-connector_2.10:1.4.1