parquet.Preconditions.checkState(ZLjava/lang/String;)V :java.lang.NoSuchMethodError

Question

运行Spark1.3.1和1.4.1时得到以下错误

parquet.Preconditions.checkState(ZLjava/lang/String；：java.lang.NoSuchMethodError( parquet.schema.Types$PrimitiveBuilder.build(Types.java:314) at parquet.schema.Types$PrimitiveBuilder.build(Types.java:232) at parquet.schema.Types$Builder.named(Types.java:210) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$1.apply(ParquetTypes.scala:314) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$fromDataType$1.apply(ParquetTypes.scala:305) at scala.Option.map(Option.scala:145)在org.apache.spark.sql.parquet.ParquetTypesConverter$.fromDataType(ParquetTypes.scala:305) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:395) at org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$4.apply(ParquetTypes.scala:394) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List。scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.parquet.ParquetTypesConverter$.convertFromAttributes(ParquetTypes.scala:393) at org.apache.spark.sql.parquet.ParquetTypesConverter$.writeMetaData(ParquetTypes.scala:440) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.prepareMetadata(newParquet.scala:260) atorg.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:276) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$6.apply(newParquet.scala:269) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)在scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newParquet.scala:269) at org.apache.spark.sql.parquet.ParquetRelation2.(newParquet.scala:391) at org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:98) at org.apache.spark.sql.parquet.DefaultSource.createRelation(newParquet.scala:128) at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:240) at org.apache.spark.sql.hive.execution.CreateMetastoreDataSourceAsSelect.run(commands.scala:218) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:54) at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:54) at org.apache.spark.sql.execution.ExecutedCommand.execute(commands.scala:64) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1099) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1099) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1121) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1071) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame.scala:1037) at org.apache.spark.sql.DataFrame.saveAsTable(DataFrame( com.xurmo.ai.spark.SparkClient.dataFrameToHiveTable(SparkClient.java:128) at com.xurmo.ai.xflow.operation.sink.DataFrameToPlatformSink.push(DataFrameToPlatformSink.java:79) at com.xurmo.ai.xflow.operation.sink.ASink.operate(ASink.java:24) at com.xurmo.ai.xflow.operation.AOperation.process(AOperation.java:121) at com.xurmo.ai.xflow.flow.executor.ExecutableOp )com.xurmo.ai.xflow.flow.executor.ExecutableOp.call(ExecutableOp.java:15) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

从日志中可以清楚地看到，类加载程序正在加载该类的其他版本。有谁能告诉我其他哪个jar包有这个类吗？

Answer 1

我也有过同样的问题。lib目录中有两个jar文件，其中包含先决条件类

1.parquet-hadoop-bundle-1.6.0rc3.jar
2.parequet-hadooop-bundle-***.jar (which does not contain checkState method)

我只移除它

parequet-hadooop-bundle-***.jar 

文件来自lib目录，现在它为我工作。