在将Spark 1.6应用程序移植到Spark 2.0.2的过程中,日志中有这样的抱怨:
com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.streaming.receiver.Receiver[]
Note: To register this class use: kryo.register(org.apache.spark.streaming.receiver.Receiver[].class);对于Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[]/class,此操作失败
sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[].class")对于Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[],此操作失败
sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[]")对于Class is not registered: org.apache.spark.streaming.receiver.Receiver[],此操作失败
sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver")对于Class is not registered: org.apache.spark.streaming.receiver.Receiver[],此操作失败
sparkConf.registerKryoClasses(Array(
classOf[org.apache.spark.streaming.receiver.Receiver[_]]
))如何注册这个类?我可以向Kryo注册其他类,但不是这个类。
编辑:
在所有这些情况下,此设置都已完成:
sparkConf.set("spark.kryo.registrationRequired", "true")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
GraphXUtils.registerKryoClasses(sparkConf)发布于 2017-03-10 04:50:58
我找到了一种在这里有效的方法:Kryo serialization refuses to register class
简而言之,将classOf[org.apache.spark.streaming.receiver.Receiver[_]],更改为classOf[Array[org.apache.spark.streaming.receiver.Receiver[_]]],
https://stackoverflow.com/questions/42683553
复制相似问题