在离线星盘群集中安装图形框架包

Question

我有一个离线电火花集群(没有互联网访问)，我需要安装图形帧库。

我已经从这里中手动下载了jar，添加在$SPARK_HOME/jar/中，然后当我尝试使用它时，会得到以下错误：

error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term typesafe in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access term scalalogging in value com.typesafe,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.typesafe.
error: missing or invalid dependency detected while loading class file 'Logging.class'.
Could not access type LazyLogging in value com.slf4j,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.slf4j.

使用所有依赖项脱机安装它的正确方法是哪一种？

Answer 1

我成功地安装了图表框架。首先，我发现了图形帧依赖关系女巫：

scala-logging-api_xx-xx.jar
scala-logging-slf4j_xx-xx.jar

其中xx是scala和jar版本的适当版本。然后我把它们安装在正确的路径上。因为我在Cloudera机器中工作，所以正确的路径是：

/opt/cloudera/parcels/spark2 2/lib/spark2 2/jars/

如果您不能将它们放在集群中的这个目录中(因为您没有根权限，而且您的管理非常懒惰)，那么您可以简单地添加您的星星之交/火花-shell。

spark-submit ..... --driver-class-path /path-for-jar/  \
                   --jars /../graphframes-0.5.0-spark2.1-s_2.11.jar,/../scala-logging-slf4j_2.10-2.1.2.jar,/../scala-logging-api_2.10-2.1.2.jar

这适用于Scala。为了将图形框架用于python，您需要下载graphframes，然后通过shell下载。

#Extract JAR content
 jar xf graphframes_graphframes-0.3.0-spark2.0-s_2.11.jar
#Enter the folder
 cd graphframes
#Zip the contents
 zip graphframes.zip -r *

然后将压缩后的文件添加到您的python路径中，然后在smack-env.sh或bash_profile中添加

export PYTHONPATH=$PYTHONPATH:/..proper path/graphframes.zip:.

然后打开shell/submitting (同样使用与scala相同的参数)，导入图形框架工作正常

这个链接对于这个解决方案非常有用。