错误:对象xml不是包com.databricks.spark的成员。

Question

我正在尝试使用SBT读取XML文件，但是我在编译它时遇到了问题。

build.sbt

name:= "First Spark"
version:= "1.0"
organization := "in.goai"
scalaVersion := "2.11.8"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "com.databricks" % "spark-avro_2.10" % "2.0.1"
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.2"
resolvers += Resolver.mavenLocal

.scala文件

package in.goai.spark

import scala.xml._
import com.databricks.spark.xml
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkContext, SparkConf}

object SparkMeApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("First Spark")
    val sc = new SparkContext(conf)
    val sqlContext = new SQLContext(sc)
    val fileName = args(0)
    val df = sqlContext.read.format("com.databricks.spark.xml").option("rowTag", "book").load("fileName")
    val selectedData = df.select("title", "price")
    val d = selectedData.show
    println(s"$d")

  }
}

当我通过给出"sbt包“来编译它时，它会显示如下错误

[error] /home/hadoop/dev/first/src/main/scala/SparkMeApp.scala:4: object xml is not a member of package com.databricks.spark
[error] import com.databricks.spark.xml
[error]        ^
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 9 s, completed Sep 22, 2017 4:11:19 PM

我需要添加任何其他与xml相关的jar文件吗？请建议并请提供任何链接给我提供关于不同文件格式的jar文件的信息

Answer 1

因为在build.sbt中使用Scala2.11和Spark2.0，所以将依赖项更改为：

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.0"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.0.0"
libraryDependencies += "com.databricks" %% "spark-avro" % "3.2.0"
libraryDependencies += "com.databricks" %% "spark-xml" % "0.4.1"
libraryDependencies += "org.scala-lang.modules" %% "scala-xml" % "1.0.6"

1. 将spark-avro版本更改为3.2.0：https://github.com/databricks/spark-avro#requirements
2. 添加"com.databricks" %% "spark-xml" % "0.4.1"：https://github.com/databricks/spark-xml#scala-211
3. 将scala-xml版本更改为1.0.6，这是Scala2.11：2.11的当前版本

在代码中，删除以下导入语句：

import com.databricks.spark.xml

请注意，您的代码实际上并不使用spark-avro或scala-xml库。如果不使用这些依赖项，则从build.sbt (以及代码中的import scala.xml._语句)中删除它们。