将HSSFWorkbook写入hdfs

Question

我需要将csv文件解析为xml并将其写入hdfs。我成功地完成了第一部分，但是在写作时会出错。这是密码。

    private static void writeToXml(String inputPath, String outputPath) throws IOException, JSchException {
        Configuration configuration = new Configuration();
        configuration.set("fs.defaultFS", "hdfs://nn");
        FileSystem fileSystem = FileSystem.get(configuration);
        Path iPath = new Path(inputPath);
        Path oPath = new Path(outputPath);
        FSDataInputStream inputStream = fileSystem.open(iPath);

        BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8));

        HSSFWorkbook workbook = new HSSFWorkbook();
        HSSFSheet sheet = workbook.createSheet("Sheet");
        AtomicReference<Integer> row = new AtomicReference<>(0);

        try (Stream<String> stream = bufferedReader.lines()) {
            stream.forEach(line -> {
                Row currentRow = sheet.createRow(row.getAndSet(row.get() + 1));
                String[] nextLine = line.split(";");
                Stream.iterate(0, i -> i + 1).limit(nextLine.length).forEach(i -> {
                    currentRow.createCell(i).setCellValue(nextLine[i]);
                });
            });
            FSDataOutputStream outputStream = fileSystem.create(oPath);
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            workbook.write(out);
            outputStream.write(out.toByteArray());
            outputStream.flush();
        }
    }

如果出现此错误，则失败。

org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B
Caused by: java.lang.NoSuchMethodError: org.apache.commons.io.IOUtils.byteArray(I)[B
    at org.apache.commons.io.output.AbstractByteArrayOutputStream.needNewBuffer(AbstractByteArrayOutputStream.java:104)
    at org.apache.commons.io.output.UnsynchronizedByteArrayOutputStream.<init>(UnsynchronizedByteArrayOutputStream.java:51)
    at org.apache.poi.poifs.filesystem.POIFSFileSystem.syncWithDataSource(POIFSFileSystem.java:779)
    at org.apache.poi.poifs.filesystem.POIFSFileSystem.writeFilesystem(POIFSFileSystem.java:756)
    at org.apache.poi.hssf.usermodel.HSSFWorkbook.write(HSSFWorkbook.java:1387)
    at path.to.package.Main.writeToXml(Main.java:81)
    at path.to.package.Main.main(Main.java:24)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:57)
    ... 15 more

在这条线。

workbook.write(out);

在我尝试写的地方编辑另一个片段。同样的错误失败。

    FSDataOutputStream outputStream = fileSystem.create(oPath);
    workbook.write(outputStream);
    outputStream.flush();

我做错了什么？

Answer 1

最后，我找不到我的依赖项有什么问题。我用下面的条件重写了这整件事。

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-hive_2.11</artifactId>
        <version>2.4.0</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>com.crealytics</groupId>
        <artifactId>spark-excel_2.11</artifactId>
        <version>0.13.7</version>
    </dependency>
    <dependency>
        <groupId>org.apache.xmlbeans</groupId>
        <artifactId>xmlbeans</artifactId>
        <version>3.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml-schemas</artifactId>
        <version>4.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.14.1</version>
    </dependency>
</dependencies>

注意，xmlbean是3.1.0版。后来的版本由于某种原因而不适用于我。

请注意，我只需要星火-excel_2.11进行本地测试。其他的添加是因为在集群上运行时，它一直与noClassDefFound一起下降。

最后，代码也变得更加简单了。

spark
  .read
  .option("delimiter", ";")
  .csv("/hdfs/path/to/file.csv")
  .repartition(1)
  .write
  .format("com.crealytics.spark.excel")
  .option("header", "false")
  .mode("overwrite")
  .save("/hdfs/path/to/file.xml")