geomesa -无法使用geomesa pyspark初始化spark sql会话

Question

我正在尝试为pyspark安装geomesa，并在初始化时收到错误

命令:geomesa_pyspark.init_sql(火花)

~/opt/anaconda3/envs/geomesa-pyspark/lib/python3.7/site-packages/geomesa_pyspark/__init__.py in init_sql(spark)
    113 
    114 def init_sql(spark):
--> 115     spark._jvm.org.apache.spark.sql.SQLTypes.init(spark._jwrapped)

TypeError: 'JavaPackage' object is not callable

我已经使用以下代码进行了安装：

pyspark == 2.4.8
geomesa_pyspark using https://repo.eclipse.org/content/repositories/geomesa-releases/org/locationtech/geomesa/
geomesa_pyspark-2.4.0.tar.gz
geomesa-accumulo-spark-runtime_2.11-2.4.0.jar
python 3.7

import geomesa_pyspark
conf = geomesa_pyspark.configure(
    jars=['./jars/geomesa-accumulo-spark-runtime_2.11-2.4.0.jar', './jars/postgresql-42.3.1.jar', './jars/geomesa-spark-sql_2.11-2.4.0.jar'],
    packages=['geomesa_pyspark','pytz'],
    spark_home='/Users/user/opt/anaconda3/envs/geomesa-pyspark/lib/python3.7/site-packages/pyspark').\
    setAppName('MyTestApp')


spark = ( SparkSession
    .builder
    .config(conf=conf) 
    .config('spark.driver.memory', '15g')
    .config('spark.executor.memory', '15g')
    .config('spark.default.parallelism', '10')
    .config('spark.sql.shuffle.partitions', '10')
    .master("local")
    .getOrCreate()
)

Answer 1

我替换了

jars=['./jars/geomesa-accumulo-spark-runtime_2.11-2.4.0.jar', './jars/postgresql-42.3.1.jar', './jars/geomesa-spark-sql_2.11-2.4.0.jar'],

至

jars=['./jars/geomesa-accumulo-spark-runtime_2.11-2.4.0.jar'],

对于postgresql，我在通过pyspark加载数据时传递了.option("driver", "org.postgresql.Driver")，这解决了这个问题。