native-lzo不可用错误| Windows 10 | Java

Question

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:61)
Caused by: java.lang.RuntimeException: native-lzo library not available
        at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
        at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)
        at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.<init>(CodecFactory.java:144)
        at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:206)
        at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:189)
        at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:287)
        at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:569)

我正在尝试在我的parquetwriter中使用lzo压缩编解码器。我不需要Hadoop，所以我的Hadoop_Home路径中只有hadoop.dll和Hadoop。在我的windows系统中，我尝试构建文档中给出的lzo2.dll，但是不知道如何使用它。我浏览了twitter fork hadoop-lzo，但windows版的使用说明并不清楚。请告诉我如何在Windows中运行此编解码器。我下载了hadoop-3.2.1，lzo-2.1.0，hadoop-lzo。

在hadoop-lzo中使用mvn全新安装会出现gplcompression.dll not found错误。

[exec] "P:\hadoop\hadoop-lzo\src\main\native\gplcompression.sln" (default target) (1) ->
     [exec] "P:\hadoop\hadoop-lzo\src\main\native\gplcompression.vcxproj.metaproj" (default target) (2) ->
     [exec] (Build target) -> 
     [exec]   P:\hadoop\hadoop-lzo\src\main\native\gplcompression.vcxproj.metaproj : warning MSB4078: The project file "gplcompression.vcxproj" is not supported by MSBuild and cannot be built.
     [exec] 
     [exec]     1 Warning(s)
     [exec]     0 Error(s)
     [exec] 
     [exec] Time Elapsed 00:00:01.57
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15.603 s
[INFO] Finished at: 2020-05-28T17:16:24+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (build-native-win) on project hadoop-lzo: An Ant BuildException has occured: Warning: Could not find file P:\hadoop\hadoop-lzo\target\native\Windows_NT-${env.PLATFORM}\gplcompression.dll to copy.
[ERROR] around Ant part ...<copy file="${build.native}/gplcompression.dll" todir="${build.native}/lib"/>... @ 19:80 in P:\hadoop\hadoop-lzo\target\antrun\build-build-native-win.xml
[ERROR] -> [Help 1]

Answer 1

它在Windows中不起作用。如果你试图为windows构建Hadoop-LZO，你在每一步都会得到构建错误。出现这个问题是因为我没有包含Hadoop-LZO (twitter)的正确构建。LZO有一些本机(系统相关)代码和许可问题，所以您需要按照"https://github.com/twitter/hadoop-lzo“中的说明从"https://www.oberhumer.com/opensource/lzo/”使用LZO构建一个特定的Hadoop-LZO。我为linux构建了它，并在linux上运行了代码，它工作了。我不能成功地为Windows构建它。我遇到了各种各样的错误，我认为Hadoop-LZO中的Windows原生代码存在问题。