无法使用Bahir启动Spark应用程序

Question

我正在尝试在Scala中运行一个Spark应用程序来连接到ActiveMQ。出于这个目的，我正在使用Bahir format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSourceProvider")。当我在我的built.sbt中使用Bahir2.2时，应用程序运行正常，但在将其更改为Bahir3.0或Bahir4.0时，应用程序无法启动，并且它给出一个错误：

[error] (run-main-0) java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream`

如何解决这个问题？我可以在我的Spark-Structured-Streaming中使用Bahir的替代方案来连接到ActiveMQ主题吗？

编辑: my build.sbt

//For spark
libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.4.0" ,
    "org.apache.spark" %% "spark-mllib" % "2.4.0" ,
    "org.apache.spark" %% "spark-sql" % "2.4.0" ,
    "org.apache.spark" %% "spark-hive" % "2.4.0" ,
    "org.apache.spark" %% "spark-streaming" % "2.4.0" ,
    "org.apache.spark" %% "spark-graphx" % "2.4.0",
)

//Bahir
libraryDependencies += "org.apache.bahir" %% "spark-sql-streaming-mqtt" % "2.4.0"

Answer 1

好的，所以看起来spark2.4和bahir2.4之间存在某种兼容性问题。我通过将它们都回滚到ver 2.3修复了它。

这是我的build.sbt

name := "sparkTest"

version := "0.1"

scalaVersion := "2.11.11"

//For spark
libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "2.3.0" ,
    "org.apache.spark" %% "spark-mllib" % "2.3.0" ,
    "org.apache.spark" %% "spark-sql" % "2.3.0" ,
    "org.apache.spark" %% "spark-hive" % "2.3.0" ,
    "org.apache.spark" %% "spark-streaming" % "2.3.0" ,
    "org.apache.spark" %% "spark-graphx" % "2.3.0",
//    "org.apache.spark" %% "spark-streaming-kafka" % "1.6.3",
)

//Bahir
libraryDependencies += "org.apache.bahir" %% "spark-sql-streaming-mqtt" % "2.3.0"