Java中将JSON转换为parquet

Question

我正在尝试将JSON转换为Java中的parquet格式，但我得到了一个异常。

输入JSON：

{"list": [ {"mainBearingX": 0.178334,
            "gearBoxZ": 0.03885,
            "_t": 1560305236290000,
            "mainBearingZ": 0.034438,
            "gearBoxX": 0.035738,
            "mainBearingY": 0.029445,
            "gearBoxY": 0.040929,
            "generatorX": 0.776837,
            "generatorY": 0.124234,
            "ts_id":"t1"
           },
           {"mainBearingX": 0.169478,
            "gearBoxZ": 0.008242,
            "_t": 1560305236311000,
            "mainBearingZ": 0.007531,
            "gearBoxX": 0.025647,
            "mainBearingY": 0.029445,
            "gearBoxY": 0.026282,
            "generatorX": 0.770189,
            "generatorY": 0.117464,
            "ts_id": "t1"
           }
         ]
}

代码：

public static void toConvert(OutPut output) {
            String inputFile = "test.parquetFile";
            Path dataFile = new Path(inputFile);
            Schema schema = ReflectData.AllowNull.get().getSchema(OutPut.class);
            try (ParquetWriter<OutPut> writer = AvroParquetWriter.<OutPut>builder(dataFile)
                    .withSchema(schema)
                    .withDataModel(ReflectData.get())
                    .withConf(new Configuration())
                    .withCompressionCodec(CompressionCodecName.SNAPPY)
                    .withWriteMode(Mode.OVERWRITE)
                    .build()) {
            } catch (IOException e) {
                e.printStackTrace();
            }
public class OutPut {
    List<Map<String, Object>> list;
} 

例外情况是：

Exception in thread "main" org.apache.parquet.schema.InvalidSchemaException: Cannot write a schema with an empty group: required group value {}
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:27)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.GroupType.accept(GroupType.java:226)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:31)
at org.apache.parquet.schema.TypeUtil$1.visit(TypeUtil.java:37)
at org.apache.parquet.schema.MessageType.accept(MessageType.java:55)
at org.apache.parquet.schema.TypeUtil.checkValidWriteSchema(TypeUtil.java:23)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:228)
at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:273)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:494)

Answer 1

问题是您的OutPut类型包含类型Object作为Map中的值类型

public class OutPut {
    List<Map<String, Object>> list;
}

您正在使用ReflectData通过内省来推断您的类型的Avro模式。但是，它不能从Object类型推断出任何有用的东西。

如果您将OutPut的定义更改为使用具体类型，例如：

public class OutPut {
    List<Map<String, Double>> list;
}

那么它应该是有效的。