SBT无法导入Kafka编解码类

Question

项目设置：

• 1生产者-序列化对象&向Kafka发送字节。
• 1火花使用者-应该在DefaultDecoder包中使用kafka.serializer来消耗字节。

发行：

• SBT导入正确的库(kafka-客户端+ kafka_2.10)，但无法在kafka_2.10 jar中找到任何类。
• 它似乎是在错误的路径下搜索(org.apache.spark.streaming.kafka而不是org.apache.kafka)。

错误消息：

    object serializer is not a member of package org.apache.spark.streaming.kafka [error] 
import kafka.serializer.DefaultDecoder.

sbt树

    [info]   +-org.apache.spark:spark-streaming-kafka_2.10:1.6.1
    [info]   | +-org.apache.kafka:kafka_2.10:0.8.2.1 [S] <-- **DefaultDecoder is in here 
but SBT can't find it (org.apache.kafka.serialization.DefaultDecoder)**
    [info]   | | +-org.apache.kafka:kafka-clients:0.8.2.1

built.sbt:

  lazy val commonSettings = Seq(
  organization := "org.RssReaderDemo",
  version := "0.1.0",
  scalaVersion := "2.10.6"
)

resolvers += "Artima Maven Repository" at "http://repo.artima.com/releases"

val spark = "org.apache.spark" % "spark-core_2.10" % "1.6.1"
val sparkStreaming = "org.apache.spark" % "spark-streaming_2.10" % "1.6.1"
val sparkStreamKafka = "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1"

// Needed to be able to parse the generated avro JSON schema
val jacksonMapperAsl = "org.codehaus.jackson" % "jackson-mapper-asl" % "1.9.13"

val scalactic = "org.scalactic" %% "scalactic" % "2.2.6"
val scalatest = "org.scalatest" %% "scalatest" % "2.2.6" % "test"

val avro = "org.apache.avro" % "avro" % "1.8.0"

lazy val root = (project in file(".")).
  settings(commonSettings: _*).
  settings(
    libraryDependencies += spark,
    libraryDependencies += sparkStreaming,
    libraryDependencies += sparkStreamKafka,
    libraryDependencies += jacksonMapperAsl,
    libraryDependencies += scalactic,
    libraryDependencies += scalatest,
    libraryDependencies += avro
  )

Answer 1

这和SBT无关。你可能有这样的东西

import org.apache.spark.streaming._
import kafka.serializer.DefaultDecoder

由于存在org.apache.spark.streaming.kafka包，因此此导入解析为org.apache.spark.streaming.kafka.serializer.DefaultDecoder。您可以导入正确的类如下：import _root_.kafka.serializer.DefaultDecoder。有关Scala的更多细节，请参见https://wiki.scala-lang.org/display/SYGN/Language+FAQs#LanguageFAQs-HowdoIimport。

Answer 2

您需要在“导入kafka.serializer.StringDecoder”之前“导入org.apache.spark.streaming._”。导入顺序可以解决问题。

工程-

import kafka.serializer.StringDecoder
import org.apache.spark.streaming._

例外-

import org.apache.spark.streaming._
import kafka.serializer.StringDecoder