我正在尝试用pyspark查询一个Elasticsearch索引,但没有成功:
] ./bin/pyspark --driver-class-path=jars/elasticsearch-hadoop-2.2.0.jar在ipython中,spark版本2.0.1:
In [1]: es_read_conf = { "es.resource" : "test/docs" , "es.nodes" : ["xx.xx.xx.aa","xx.xx.xx.bb","xx.xx.xx.cc"],"es.port" : "9200", "es.net.http.auth.user": "myusername", "es.net.http.auth.pass": "mypassword"}
es_rdd = sc.newAPIHadoopRDD(inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",keyClass="org.apache.hadoop.io.NullWritable", valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable", conf=es_read_conf)我得到以下错误:
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.
: java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.lang.String似乎在将es.nodes的python列表转换为Java string时出现了问题。我尝试使用一个只包含elasticsearch主节点地址(“xx.aa”)的字符串,但得到另一个错误:
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.
: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [test/docs] failed; server[xx.xx.xx.bb:9202] returned [502|Bad Gateway:]有时错误指的是数据节点bb,有时指的是cc。有趣的是,如果我多次运行同一命令,就不会得到任何错误(可能是在仅针对主节点运行查询时?)。将本地主机作为唯一的es.nodes运行该命令没有任何问题。
发布于 2017-01-24 14:22:01
https://stackoverflow.com/questions/41809985
复制相似问题