如何在IDE上的Scala项目中利用Delta湖

Question

我在我的build.sbt中添加了增量依赖项

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-hive" % sparkVersion,
  // logging
  "org.apache.logging.log4j" % "log4j-api" % "2.4.1",
  "org.apache.logging.log4j" % "log4j-core" % "2.4.1",
  // postgres for DB connectivity
  "org.postgresql" % "postgresql" % postgresVersion,
  "io.delta" %% "delta-core" % "0.7.0"

但是，我不知道火花会话必须包含什么配置。下面的代码失败。

val spark = SparkSession.builder()
    .appName("Spark SQL Practice")
    .config("spark.master", "local")
    .config("spark.network.timeout"  , "10000000s")//to avoid Heartbeat exception
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog")
    .getOrCreate()

例外-

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/plans/logical/MergeIntoTable

Answer 1

这是我做的一个例子，它能帮你。

build.sbt文件应该包括以下这些依赖项：

libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.0.0" % "provided"
libraryDependencies += "io.delta" %% "delta-core" % "0.7.0" % "provided"

我认为你需要成为将星火3号用于三角洲湖0.7.0。

您不应该需要任何特殊的SparkSession配置选项，这样的东西应该可以：

lazy val spark: SparkSession = {
  SparkSession
    .builder()
    .master("local")
    .appName("spark session")
    .config("spark.databricks.delta.retentionDurationCheck.enabled", "false")
    .getOrCreate()
}

Answer 2

这是在您的代码所依赖的类文件在编译时存在但在运行时没有找到时造成的。查找构建时间和运行时类路径的差异。

更具体到您的场景：

If you get  java.lang.NoClassDefFoundError on
org/apache/spark/sql/catalyst/plans/logical/MergeIntoTable exception 
in this case JAR version does not have MergeIntoTable.scala file. 

The solution was to add the apache spark latest version, which comes with the
org/apache/spark/sql/catalyst/plans/logical/MergeIntoTable.scala file . 

更多信息，在火花3.x.x升级和发布- https://github.com/apache/spark/pull/26167。

Answer 3

你需要升级阿帕奇星火。在v3.0.0版本中引入了MergeIntoTable特性。链接到源代码：AstBuilder.scala、Analyzer.scala、Github拉出请求、发行说明 (查看功能增强部分)。