Scala : java.lang.NoClassDefFoundError: org/apache/log4j/LogManager线程中的异常

Question

我是Scala的初学者，我试图在scala中运行一个模型，但是我面临一些问题：

这是一个文件：

package com.salesforce.hw.titanic

import com.salesforce.op._
import com.salesforce.op.features.FeatureBuilder
import com.salesforce.op.features.types._
import com.salesforce.op.readers.DataReaders
import com.salesforce.op.stages.impl.classification._
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Level, LogManager}



/**
 * A minimal Titanic Survival example with TransmogrifAI
 */



object OpTitanicMini {

  case class Passenger
  (
    id: Long,
    survived: Double,
    pClass: Option[Long],
    name: Option[String],
    sex: Option[String],
    age: Option[Double],
    sibSp: Option[Long],
    parCh: Option[Long],
    ticket: Option[String],
    fare: Option[Double],
    cabin: Option[String],
    embarked: Option[String]
  )



  def main(args: Array[String]): Unit = {
    LogManager.getLogger("com.salesforce.op").setLevel(Level.ERROR)
    implicit val spark = SparkSession.builder.config(new SparkConf()).getOrCreate()
    import spark.implicits._

    // Read Titanic data as a DataFrame
    val pathToData = Option(args(0))
    val passengersData = DataReaders.Simple.csvCase[Passenger](pathToData, key = _.id.toString).readDataset().toDF()

    // Automated feature engineering
    val (survived, features) = FeatureBuilder.fromDataFrame[RealNN](passengersData, response = "survived")
    val featureVector = features.toSeq.autoTransform()

    // Automated feature selection
    val checkedFeatures = survived.sanityCheck(featureVector, checkSample = 1.0, removeBadFeatures = true)

    // Automated model selection
    val (pred, raw, prob) = BinaryClassificationModelSelector().setInput(survived, checkedFeatures).getOutput()
    val model = new OpWorkflow().setInputDataset(passengersData).setResultFeatures(pred).train()

    println("Model summary:\n" + model.summaryPretty())








  }

}

当我试图运行它时，我会得到以下错误：

线程"main“中的异常: org/apache/log4j/LogManager at com.salesforce.hw.titanic.OpTitanicMini$.main(OpTitanicMini.scala:72) at com.salesforce.hw.titanic.OpTitanicMini.main(OpTitanicMini.scala)，由: java.lang.ClassNotFoundException: org.apache.log4j.LogManager at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349引起)在java.lang.ClassLoader.loadClass(ClassLoader.java:357) .还有2

我试着看了一下这个问题，发现了这个博客帖子，我试了一下博客上说的话：

我的log4j.properties文件看起来如下：

log4j.rootCategory=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Settings to quiet third party logs that are too verbose
log4j.logger.Remoting=ERROR
log4j.logger.org.eclipse.jetty=ERROR
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

# Change this to set Hadoop log level
log4j.logger.org.apache.hadoop=ERROR

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Change this to set Spark log level
log4j.logger.org.apache.spark=ERROR

# Breeze
log4j.logger.breeze.optimize=FATAL

# BLAS & LAPACK
log4j.logger.com.github.fommil.netlib=ERROR

# TransmogrifAI logging
log4j.logger.com.salesforce.op=INFO
log4j.logger.com.salesforce.op.utils.spark.OpSparkListener=OFF

# Helloworld logging
log4j.logger.com.salesforce.hw=INFO

我尝试了博客文章中提到的步骤，但仍然面临同样的问题，我如何解决这个问题？

Answer 1

LogManager类附带一个星火依赖项。确保在运行时拥有org.apache.spark:spark-core、org.apache.spark:spark-mlib、org.apache.spark:spark-sql以及它们在类路径上的所有传递依赖项。

我们有一个示例sbt项目这里，您可以查看它。