安装栅格框的Java错误(数据库)

Question

我遵循在这个笔记本上的步骤在我的databricks集群上安装了栅格框架。

最后，我能够导入以下内容：

​

from pyrasterframes import rf_ipython
from pyrasterframes.utils import create_rf_spark_session
from pyspark.sql.functions import lit 
from pyrasterframes.rasterfunctions import *

​

但当我跑的时候

​

spark = create_rf_spark_session()

​

我得到了以下错误："java.lang.NoClassDefFoundError: scala/Product$class“。

我使用的是星火3.2.1的集群。我还安装了1.8.0_341，但这并没有什么区别。

有人能解释一下出了什么问题吗？如何解决这个错误？

完整的错误日志：

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<command-2354681519525034> in <module>
      5 
      6 # Use the provided convenience function to create a basic local SparkContext
----> 7 spark = create_rf_spark_session()
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/utils.py in create_rf_spark_session(master, **kwargs)
     97 
     98     try:
---> 99         spark.withRasterFrames()
    100         return spark
    101     except TypeError as te:
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/__init__.py in _rf_init(spark_session)
     42     """ Adds RasterFrames functionality to PySpark session."""
     43     if not hasattr(spark_session, "rasterframes"):
---> 44         spark_session.rasterframes = RFContext(spark_session)
     45         spark_session.sparkContext._rf_context = spark_session.rasterframes
     46 
 
/databricks/python/lib/python3.8/site-packages/pyrasterframes/rf_context.py in __init__(self, spark_session)
     37         self._jvm = self._gateway.jvm
     38         jsess = self._spark_session._jsparkSession
---> 39         self._jrfctx = self._jvm.org.locationtech.rasterframes.py.PyRFContext(jsess)
     40 
     41     def list_to_seq(self, py_list):
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1566 
   1567         answer = self._gateway_client.send_command(command)
-> 1568         return_value = get_return_value(
   1569             answer, self._gateway_client, None, self._fqn)
   1570 
 
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)
 
/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)
 
Py4JJavaError: An error occurred while calling None.org.locationtech.rasterframes.py.PyRFContext.
: java.lang.NoClassDefFoundError: scala/Product$class
    at org.locationtech.rasterframes.model.TileDimensions.<init>(TileDimensions.scala:35)
    at org.locationtech.rasterframes.package$.<init>(rasterframes.scala:55)
    at org.locationtech.rasterframes.package$.<clinit>(rasterframes.scala)
    at org.locationtech.rasterframes.py.PyRFContext.<init>(PyRFContext.scala:49)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:250)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:251)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: scala.Product$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
    at com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
    ... 15 more

​

提前感谢你？

Answer 1

该版本的RasterFrames (0.8.4)只适用于使用Spark2.4和Scala2.11的DBR6.x，而不会在使用Scala2.12的Spark3.2.x上工作。您可以尝试使用0.10.1版本，而不是升级到Spark3.1.2，但它可能无法使用Spark3.2(我还没有对它进行测试)。

如果您想在Databricks上执行地理空间查询，可以从Databricks实验室查看Mosaic项目--它支持标准的st_函数&许多其他东西。您可以在下面的博客帖子中找到公告，更多信息在谈数据与人工智能峰会2022年，文档 & GitHub项目中。