NettyBlockTransferService不尊重spark.blockManager.port配置

Question

我在运行星火2.4.4。在纱线上。NodeManagers上的火花配置如下所示：

spark-defaults.conf:

spark.driver.port=38429
spark.blockManager.port=35430
spark.driver.blockManager.port=44349

当创建火花驱动程序和执行器时，它们会选择驱动端口(38429)配置，而不是blockManager (35430) / driver.blockManager (44349)配置。blockManager端口被随机分配

司机：

14:23:40 INFO spark.SparkContext: Running Spark version 2.4.4
14:23:40 INFO util.Utils: Successfully started service 'sparkDriver' on port **38429**.
14:23:41 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38171.
14:23:41 INFO netty.NettyBlockTransferService: Server created on driverhost:**38171**

遗嘱执行人：

14:23:44 INFO client.TransportClientFactory: Successfully created connection to driverhost:**38429** after 73 ms (0 ms spent in bootstraps)
14:23:45 INFO executor.Executor: Starting executor ID 1 on host ...
14:23:45 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34914.
14:23:45 INFO netty.NettyBlockTransferService: Server created on executorhost:**34914**

我遇到了一个描述这个问题的bug Jira，但它是针对Spark2.4.0提出的，12个月前关闭了：https://issues.apache.org/jira/browse/SPARK-27139

看一下GitHub中的火花代码，我找不到任何明显的东西：

https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/SparkEnv.scala

333    val blockManagerPort = if (isDriver) {
334      conf.get(DRIVER_BLOCK_MANAGER_PORT)
335    } else {
336      conf.get(BLOCK_MANAGER_PORT)
337    }
338
339    val blockTransferService =
340      new NettyBlockTransferService(conf, securityManager, bindAddress, advertiseAddress,
341        blockManagerPort, numUsableCores)

https://github.com/apache/spark/blob/branch-2.4/core/src/main/scala/org/apache/spark/internal/config/package.scala

308  private[spark] val BLOCK_MANAGER_PORT = ConfigBuilder("spark.blockManager.port")
309    .doc("Port to use for the block manager when a more specific setting is not provided.")
310    .intConf
311    .createWithDefault(0)
312
313  private[spark] val DRIVER_BLOCK_MANAGER_PORT = ConfigBuilder("spark.driver.blockManager.port")
314    .doc("Port to use for the block manager on the driver.")
315    .fallbackConf(BLOCK_MANAGER_PORT)

有人能告诉我为什么我的NettyBlockTransferService端口是随机分配的，而不是35430或44349？

Answer 1

这里的问题是在纱线NodeManagers上设置这个配置。它需要在客户机上设置，即提交Spark应用程序的进程，而不是集群本身。