Azure VM中的火花主IP配置

Question

我将在Azure中建立独立的Spark集群。我希望使用Azure VM的公共IP运行Spark主程序，而不是使用VM的主机名，这样我就可以从其他VM访问。

火花版本: spark-2.2.0-bin-hadoop2.7

我已经在conf文件夹下创建了新的文件“smack-env.sh”，并添加了导出SPARK_MASTER_HOST=x.x。

Start master sbin>./start-master.sbin

我得到了下面提到的错误。火花大师还没有启动。

如何将公共IP地址设置为Spark ?

错误日志

18/04/10 04:55:12 INFO SecurityManager: Changing view acls to: root
18/04/10 04:55:12 INFO SecurityManager: Changing modify acls to: root
18/04/10 04:55:12 INFO SecurityManager: Changing view acls groups to:
18/04/10 04:55:12 INFO SecurityManager: Changing modify acls groups to:
18/04/10 04:55:12 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7082. Attempting port 7083.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7083. Attempting port 7084.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7084. Attempting port 7085.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7085. Attempting port 7086.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7086. Attempting port 7087.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7087. Attempting port 7088.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7088. Attempting port 7089.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7089. Attempting port 7090.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7090. Attempting port 7091.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7091. Attempting port 7092.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7092. Attempting port 7093.
18/04/10 04:55:13 WARN Utils: Service 'sparkMaster' could not bind on port 7092. Attempting port 7093.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)! Consider explicitly setting the appropriate port for the service 'sparkMaster' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)

Answer 1

对于SPARK_MASTER_HOST，您应该使用VM的私有IP或0.0.0.0，您不能输入VM的公共IP，如果您这样做，您将得到错误日志。

现在，您希望通过公共IP访问Spark，您需要Azure NSG和VM的防火墙上的开放端口Azure NSG(不确定，取决于您的服务)。