从EMR火花连接到EMR预置-连接失败

Question

我遇到了问题，从一个AWS EMR集群连接到另一个运行前的AWS EMR集群。

用python编写的代码是：

jdbcDF = spark.read \
        .format("jdbc") \
        .option("driver", "com.facebook.presto.jdbc.PrestoDriver")\
        .option("url", "jdbc:presto://ec2-xxxxxxxxxxxx.ap-southeast-2.compute.amazonaws.com:8889/hive/data-lake") \
        .option("user", "hadoop") \
        .option("dbtable", "customer") \
        .load()\

通过aws emr add-steps与选项--packages,\'org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.0,org.postgresql:postgresql:42.2.9,com.facebook.presto:presto-jdbc:0.60\',\进行部署

在部署时抛出以下错误

线程"main“java.lang.reflect.UndeclaredThrowableException中的

异常在org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:237) at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:330) at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)由: org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)：org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101) at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:250) at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:引发的awaitResult: at awaitResult: at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:引发的异常65)在org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64) at java.security.AccessController.doPrivileged(Native Method)在javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) .还有4条原因是: java.io.IOException:未能连接到ip-xxxx-xxx.ap-southeast-2.compute.internal/xxx-xxxx：41885在org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187) at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194) at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)由: io.netty.channel.AbstractChannel$AnnotatedConnectException:连接被拒绝: ip-xxxxxxxxx.ap-southeast-2.compute.internal/xxxxxx:41885 at sun.nio.ch.SocketChannelImpl.checkConnect(Native方法引起)在sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.在io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) .1多一个原因是: java.net.ConnectException:连接被拒绝.LogType的另一端:stderr

虽然我已经编辑了上面的IP地址(安全第一)，但它是火花从实例上相同的内部IP地址。它似乎正在连接到自己，并且存在连接问题。

我已经打开了AWS EC2安全组中的端口，允许从spark主/从访问presto实例。

如果有帮助，编写用于测试连接性的快速节点脚本可以工作。

var client = new presto.Client({
  host: prestoEndpoint,
  user: 'hadoop',
  port: 8889,
});

client.execute({
  query: 'select * from customer',
  catalog: 'hive',
  schema: 'data-lake',
  source: 'nodejs-client',
  state: function(error, query_id, stats) {
     console.log({ message: 'status changed', id: query_id, stats: stats });
  },
  columns: function(error, data) {
     console.log({ resultColumns: data });
  },
  data: function(error, data, columns, stats) {
    console.log({data, columns});
  },
  success: function(error, stats) {
     console.log(error);
     console.log(JSON.stringify(stats, null,2));
  },
  error: function(error) {
    console.log(error);
  },
});

错误消息的关键部分似乎是

由: io.netty.channel.AbstractChannel$AnnotatedConnectException:连接拒绝: ip-xxxxxxxxx.ap-southeast-2.compute.internal/xxxxxx:41885引起的

Answer 1

问题是prest-jdbc驱动程序的版本号。

我将它从com.facebook.presto:presto-jdbc:0.60更新为com.facebook.presto:presto-jdbc:0.225，所以完整的包参数是

--packages,\'org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.0,org.postgresql:postgresql:42.2.9,com.facebook.presto:presto-jdbc:0.255\',\

感谢@Lamanus发现了那个