Qubole#火花透镜；0.3.1-S2.11:找不到(电火花EMR)

Question

我试图通过电子病历执行火花镜头，并得到这个错误，下面是我的火花提交命令。

我使用的是pyspark 3.0.1和hadoop 3.1.1节。

spark-submit --packages Qubole:sparklens:0.3.1-s_2.11 --conf Spark.extraListeners=com.qubole.Sparklens.QuboleJobListener --py-files config.zip,jobs.zip,DDL.zip main.py

我得到了

问题摘要:未找到警告模块:qubole#信号灯；0.3.0-s_2.11

==== local-m2-cache: tried

  file:/home/hadoop/.m2/repository/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  file:/home/hadoop/.m2/repository/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

==== local-ivy-cache: tried

  /home/hadoop/.ivy2/local/qubole/sparklens/0.3.0-s_2.11/ivys/ivy.xml

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  /home/hadoop/.ivy2/local/qubole/sparklens/0.3.0-s_2.11/jars/sparklens.jar

==== central: tried

  https://repo1.maven.org/maven2/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  https://repo1.maven.org/maven2/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

==== spark-packages: tried

  https://dl.bintray.com/spark-packages/maven/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.pom

  -- artifact qubole#sparklens;0.3.0-s_2.11!sparklens.jar:

  https://dl.bintray.com/spark-packages/maven/qubole/sparklens/0.3.0-s_2.11/sparklens-0.3.0-s_2.11.jar

    ::::::::::::::::::::::::::::::::::::::::::::::

    ::          UNRESOLVED DEPENDENCIES         ::

    ::::::::::::::::::::::::::::::::::::::::::::::

    :: qubole#sparklens;0.3.0-s_2.11: not found

    ::::::::::::::::::::::::::::::::::::::::::::::

谢了，习

Answer 1

看起来，sparklens模块在任何火花存储库中都找不到。

解决方案之一是从sparklens-0.3.0-s_2.11.jar下载jar文件，并使用选项--jars代替--packages，如下所示：

spark-submit --jars ./sparklens-0.3.0-s_2.11.jar ...

(假设jar位于当前文件夹中)

或者，您可以使用--repositories选项指定一个不同的存储库，但我没有找到一个工作存储库https://repos.spark-packages.org和bintray.com都没有工作)