如何使用apache-spark引导读取CSV文件

Question

我是Apache的新手，我正在尝试使用从Apache读取CSV文件。然而，尽管尝试了几种不同的方法，但我仍然得到了下面的异常。

是否需要额外的配置？

SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark SQL Example")
                .getOrCreate();
    
    
    StructType schema = new StructType()
            .add("department", "string")
            .add("designation", "string")
            .add("ctc", "long")
            .add("state", "string");
    
    
    Dataset<Row> df = spark.read()
            .option("mode", "DROPMALFORMED")
            .schema(schema)
            .csv("C://exampl/bands-csv.csv");



java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
    at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:547) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:568) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:591) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:688) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) ~[hadoop-client-api-3.3.2.jar:na]
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
    at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:467) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:438) ~[hadoop-client-api-3.3.2.jar:na]
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:515) ~[hadoop-client-api-3.3.2.jar:na]
    ... 21 common frames omitted

2022-10-10 09:51:44.030  WARN 20884 --- [  restartedMain] org.apache.hadoop.util.NativeCodeLoader  : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-10-10 09:51:44.168 ERROR 20884 --- [  restartedMain] org.apache.spark.SparkContext            : Error initializing SparkContext.

org.apache.spark.SparkException: A master URL must be set in your configuration
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:396) ~[spark-core_2.13-3.3.0.jar:3.3.0]
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704) ~[spark-core_2.13-3.3.0.jar:3.3.0]
    at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953) ~[spark-sql_2.13-3.3.0.jar:3.3.0]
    at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.8.jar:na]
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947) ~[spark-sql_2.13-3.3.0.jar:3.3.0]
    at com.spark.sql.SparkPocApplication.main(SparkPocApplication.java:42) ~[classes/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.7.4.jar:2.7.4]

2022-10-10 09:51:44.229  INFO 20884 --- [  restartedMain] org.apache.spark.SparkContext            : Successfully stopped SparkContext
Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

依赖项添加了apche火花核心和apache火花sql。

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.13</artifactId>
    <version>3.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.13</artifactId>
    <version>3.3.0</version>
    <scope>provided</scope>
</dependency>

Answer 1

问题已经解决，它丢失了HADOOP_HOME目录和winutils 温蒂尔斯以及hadoop.dll HADOOP.DLL的配置。

解决方案和配置可以在这里找到：解决方案