火花- fpgrowth -关联规则- StackOverflow错误

Question

我有一个巨大的dataframe (500万行)，每一行都是一个项目篮，我正在尝试获取频繁的项集和关联规则。但是它给了我StackOverflowErrors，我试着设置检查点目录，但它没有解决问题。知道怎么解决这个问题吗？提前谢谢

fpGrowth = FPGrowth(itemsCol="ARFeatures", minSupport=0.8, minConfidence=0.9)

model = fpGrowth.fit(completeDf)

java.lang.reflect.InvocationTargetException.(InvocationTargetException.java:72) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStreamjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStreamjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:138) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:136) at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) at scala.collection.mutable.HashTable$class.serializeTo(HashTable.scala:125) at scala.collection.mutable.HashMap.serializeTo(HashMap.scala:40) at scala.collection.mutable.HashMap.writeObject(HashMap.scala:136) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498)在java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1028) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)在java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at scala.collection.mutable.HashMap$$anonfun$writeObject$1.apply(HashMap.scala:138)

Answer 1

增加驱动程序堆栈大小。这取决于您如何执行应用程序，您需要正确地传递驱动程序JVM选项。

对于星火提交，您可以将其添加为cmd行arg：

--conf "spark.driver.extraJavaOptions=-Xss10m"

要了解更多细节，请看这些内容：

• https://issues.apache.org/jira/browse/SPARK-18531
• How to pass -D parameter or environment variable to Spark job?