错误:发生JNI错误，请检查您的安装，然后在使用SBT的Scala-Spark的IntelliJ IDEA中再试一次

Question

import org.apache.spark.sql.SparkSession

object RDDBroadcast extends App {

  val spark = SparkSession.builder()
    .appName("SparkByExamples.com")
    .master("local")
    .getOrCreate()

  val states = Map(("NY","New York"),("CA","California"),("FL","Florida"))
  val countries = Map(("USA","United States of America"),("IN","India"))

  val broadcastStates = spark.sparkContext.broadcast(states)
  val broadcastCountries = spark.sparkContext.broadcast(countries)

  val data = Seq(("James","Smith","USA","CA"),
    ("Michael","Rose","USA","NY"),
    ("Robert","Williams","USA","CA"),
    ("Maria","Jones","USA","FL")
  )

  val rdd = spark.sparkContext.parallelize(data)

  val rdd2 = rdd.map(f=>{
    val country = f._3
    val state = f._4
    val fullCountry = broadcastCountries.value(country)
    val fullState = broadcastStates.value(state)
    (f._1,f._2,fullCountry,fullState)
  })

  println(rdd2.collect().mkString("\n"))

}

以上是星星之火-scala代码，以获取国家和州的名称。在编译InteliJ IDEA中的代码时，我得到的错误如下：

*Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 7 more*

我尝试过多种解决方案，比如检查java和javac版本，但是版本是同步的。单击此处查看图像。

我还在project >project.-> Module中检查了java版本，并将其与Run->Edit配置进行了比较，后者也是匹配的。

Build.sbt代码：单击此处查看sbt代码

我使用的是VMware工作站16播放器，inteliJ安装在linux上。java版本是1.8.0_301

单击此处查看Linux中的InteliJ IDE

Answer 1

我也遇到了同样的问题，通过将Run/Debug配置->修改选项-> Java ->添加依赖项(提供的范围)到类路径设置为YES来解决这个问题。