首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >Pyspark集成到Pycharm中

问Pyspark集成到Pycharm中
EN

Stack Overflow用户
提问于 2015-07-22 04:30:37
回答 4查看 2.1K关注 0票数 3

我不知道如何配置Pycharm,这样我就可以直接在Pyspark中运行脚本。我在Elasticsearch集群上使用Pyspark,并使用以下代码来运行脚本。当我尝试将pyspark shell配置为解释器时,它使用默认的python解释器运行,但由于错误it I not a valid SDK而无法工作:

代码语言:javascript
复制
__author__ = 'lucas'


from pyspark import SparkContext, SparkConf

if __name__ == "__main__":

    conf = SparkConf().setAppName("ESTest")
    sc = SparkContext(conf=conf)

    es_read_conf = {
        "es.nodes" : "localhost",
        "es.port" : "9200",
        "es.resource" : "titanic/passenger"
    }
    es_rdd = sc.newAPIHadoopRDD(
        inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
        keyClass="org.apache.hadoop.io.NullWritable",
        valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
        conf=es_read_conf)

    es_write_conf = {
        "es.nodes" : "localhost",
        "es.port" : "9200",
        "es.resource" : "titanic/value_counts"
    }

    doc = es_rdd.first()[1]

    for field in doc:

        value_counts = es_rdd.map(lambda item: item[1][field])
        value_counts = value_counts.map(lambda word: (word, 1))
        value_counts = value_counts.reduceByKey(lambda a, b: a+b)
        value_counts = value_counts.filter(lambda item: item[1] > 1)
        value_counts = value_counts.map(lambda item: ('key', {
            'field': field,
            'val': item[0],
            'count': item[1]
        }))

        value_counts.saveAsNewAPIHadoopFile(
            path='-',
            outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
            keyClass="org.apache.hadoop.io.NullWritable",
            valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
            conf=es_write_conf)

但这会生成以下堆栈跟踪:

代码语言:javascript
复制
Traceback (most recent call last):
  File "/home/lucas/PycharmProjects/tweetspark/analytics/tweetanalyzer.py", line 20, in <module>
    conf=es_read_conf)
  File "/var/opt/spark/python/pyspark/context.py", line 601, in newAPIHadoopRDD
    jconf, batchSize)
  File "/var/opt/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/var/opt/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.
: java.lang.ClassNotFoundException: org.elasticsearch.hadoop.mr.LinkedMapWritable
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:278)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:179)
    at org.apache.spark.api.python.PythonRDD$.newAPIHadoopRDDFromClassNames(PythonRDD.scala:519)
    at org.apache.spark.api.python.PythonRDD$.newAPIHadoopRDD(PythonRDD.scala:503)
    at org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)
EN

回答 4

Stack Overflow用户

发布于 2016-05-14 02:16:18

您缺少的是elasticsearch-spark.jar。下载ElasticSearch-Hadoop integration,在dist子目录下找到elasticsearch-spark,然后设置SPARK_CLASSPATH环境变量

代码语言:javascript
复制
os.environ['SPARK_CLASSPATH'] = "/path/to/elasticsearch-hadoop-2.3.0/dist/elasticsearch-spark_2.10-2.3.0.jar"
票数 3
EN

Stack Overflow用户

发布于 2015-07-30 12:53:53

堆栈跟踪正在抱怨一个丢失的jar。通过在启动SparkContext之前添加以下代码,您可以将其添加到类路径中

代码语言:javascript
复制
import os
os.environ['SPARK_CLASSPATH'] = "/path/to/elasticsearch-hadoop.jar"

conf = SparkConf().setAppName("ESTest")
sc = SparkContext(conf=conf)

...
票数 2
EN

Stack Overflow用户

发布于 2019-05-22 17:40:09

我正在使用pipenv和pyspark在pycharm中进行本地开发。为了不将任何指定外部jar路径的代码引入到您的项目中,您可以下载缺少的jar并将它们复制到默认的jar文件路径。

如何找到pyspark需要的默认路径,包括jar文件。

  1. 在虚拟python环境

中查找路径

代码语言:javascript
复制
$ which pyspark
/Users/xxxx/.local/share/virtualenvs/demo-spark-ZXzB9uOI/bin/pyspark

  1. 在/Users/xxxx/.local/share/virtualenvs/demo-spark-ZXzB9uOI/bin/下运行find_spark_home.py并获取spark home

的路径

代码语言:javascript
复制
$ python /Users/xxxx/.local/share/virtualenvs/demo-spark-ZXzB9uOI/bin/find_spark_home.py

/Users/xxxx/.local/share/virtualenvs/abnormal_detection-ZXzB9uOI/lib/python3.6/site-packages/pyspark

则默认路径为/Users/xxxx/.local/share/virtualenvs/abnormal_detection-ZXzB9uOI/lib/python3.6/site-packages/pyspark/jars

  1. 将外部jar文件复制到

的缺省路径

代码语言:javascript
复制
$ cp xxxx.jar /Users/xxxx/.local/share/virtualenvs/abnormal_detection-ZXzB9uOI/lib/python3.6/site-packages/pyspark/jars/

希望能对你有所帮助。

票数 2
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/31549016

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档