Spark-shell不工作

Question

当我提交spark-shell命令时，我看到以下错误：

# spark-shell

> SPARK_MAJOR_VERSION is set to 2, using Spark2
  File "/usr/bin/hdp-select", line 249
    print "Packages:"
                    ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(t "Packages:")?
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread "main" java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file in conf with -Dhdp.version=xxx
        at org.apache.spark.launcher.Main.main(Main.java:118)

Answer 1

问题是HDP脚本/usr/bin/hdp-select显然是在Python3下运行的，而它包含不兼容的Python2特定代码。

您可以通过以下方式将/usr/bin/hdp- Python3连接到select：

"list(package).sort()")

• replacing statements

• replacing "os.mkdir(current，0755)“by "os.mkdir(current，0o755)"

将括号添加到打印"packages.sort()”行

您还可以尝试强制HDP在Python2下运行/usr/bin/hdp-select：

PYSPARK_DRIVER_PYTHON=python2 PYSPARK_PYTHON=python2 spark-shell

Answer 2

我也有同样的问题:我在运行spark之前设置了HDP_VERSION。

export HDP_VERSION=<your hadoop version>

spark-shell