Flink with Guava cache - ProcessFunction的实现是不可序列化的

Question

我实现了一个ProcessFunction，它使用Guava缓存来过滤传入的事件流。代码如下所示：

object myJob {
 private def updateCache(cacheObject, someValue) = {}
 private def getCacheValue(cacheObject, someKey) = {}

 override def run(params, executionEnv) = {
  val inputStream = executionEnv.stream

  val c = CacheBuilder.newBuilder()

  val outStream = inputStream.process(new ProcessFunction() { 
    updateCache()
    getCacheValue} 
    )
 }
}

在将作业提交到Flink时，我收到以下错误：

Caused by: org.apache.flink.api.common.InvalidProgramException: The implementation of the ProcessFunction is not serializable. The object probably contains or references non serializable fields.
at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:99)
at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1560)
at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:185)
at org.apache.flink.streaming.api.datastream.DataStream.process(DataStream.java:666)
at org.apache.flink.streaming.api.scala.DataStream.process(DataStream.scala:686)

你知道我做错了什么吗？如何解决此序列化错误？

Answer 1

这个错误基本上说明你依赖的对象对于Flink是不可序列化的。在您已经展示的情况下，将带有装载器的字段标记为惰性应该可以解决这个问题：

   lazy val c = CacheBuilder.newBuilder()

通常，在这种情况下，您应该参考Flink的documentation，它解释了这个问题