使用spark-cloudant连接器从本地Spark集群(PySpark)连接Cloudant DB时出错

Question

    I am getting the following error while connecting Cloudant DB from my local Spark cluster(PySpark) using spark-cloudant connector.

    The spark-cloudant connector is: https://spark-packages.org/package/cloudant-labs/spark-cloudant

    I am started my PySpark with following command,
    pyspark2.cmd --jars "all jars for spark-cloudant connector"

The code I am using, 
sqlContextNew = SQLContext(sc)
booksDataDF = sqlContextNew.read.format("com.cloudant.spark").option("cloudant.host","AAA-HOST").option("cloudant.username", "BBB-USERNAME"). option("cloudant.password","CCC-PASSWORD").load("booksdata")

错误消息是: 16/07/22 23:01:51信息JsonStoreDataAccess: reuse SparkEnv ActorSystem，因为它包含喷雾16/07/22 23:01:52 ERROR ActorSystemImpl: Uncaught SparkEnv error from thread sparkDriverActorSystem-akka.actor.default-dispatcher-2 shutting ActorSystem sparkDriverActorSystem java.lang.ExceptionInInitializerError

原因: scala.Predef$ArrowAssoc$.extension$$minus$greater(Ljava/lang/Object;Ljava/lang/Object；：java.lang.NoSuchMethodError)Lscala/Tuple2在org.parboiled.scala.Parser$class.rule(Parser.scala:73) at org.parboiled.scala.Parser$class.rule(Parser.scala:46) at spray.http.parser.HttpParser$.rule(HttpParser.scala:29) at spray.http.parser.CORSHeaders$class.$timesAccess$minusControl$minusMax$minusAge(CORSHeaders.scala:59) at spray.http.parser.HttpParser$.$timesAccess$minusControl$minusMax$minusAge(HttpParser.scala:29)

Answer 1

我怀疑这个错误是由于错误的Scala版本造成的。您使用的scala版本是什么？对于spark和spark-cloudant，您都需要Scala 2.10。