executor.CoarseGrainedExecutorBackend (Logging.scala:logInfo(57)) -驱动程序命令关闭-我如何可以在驱动端调试？

Question

我从执行者那里获得日志(从buttom开始)：

2021-11-30 21:44:42 
2021-11-30 18:44:42,911 INFO  [shutdown-hook-0] util.ShutdownHookManager (Logging.scala:logInfo(57)) - Deleting directory /var/data/spark-0646270c-a2d0-47d4-8e6c-0bc735bc255d/spark-a54cf7e4-baaf-4411-9073-0c1fb1e4cc5b
2021-11-30 21:44:42 
2021-11-30 18:44:42,910 INFO  [shutdown-hook-0] util.ShutdownHookManager (Logging.scala:logInfo(57)) - Shutdown hook called
2021-11-30 21:44:42 
2021-11-30 18:44:42,902 ERROR [SIGTERM handler] executor.CoarseGrainedExecutorBackend (SignalUtils.scala:$anonfun$registerLogger$2(43)) - RECEIVED SIGNAL TERM
2021-11-30 21:44:42 
2021-11-30 18:44:42,823 INFO  [CoarseGrainedExecutorBackend-stop-executor] storage.BlockManager (Logging.scala:logInfo(57)) - BlockManager stopped
2021-11-30 21:44:42 
2021-11-30 18:44:42,822 INFO  [CoarseGrainedExecutorBackend-stop-executor] memory.MemoryStore (Logging.scala:logInfo(57)) - MemoryStore cleared
2021-11-30 21:44:42 
2021-11-30 18:44:42,798 INFO  [dispatcher-Executor] executor.CoarseGrainedExecutorBackend (Logging.scala:logInfo(57)) - Driver commanded a shutdown

我怎样才能让任何类型的日志记录在火花驱动程序中理解，驱动程序上的什么事件触发了执行器关闭？对于驱动程序或执行器来说，内存并不缺乏，pod指标表明它们占用的内存远远超过有限的+开销。因此，看起来关闭信号的原因并不是缺少资源，而是一些隐藏的异常，在任何地方都没有记录。

根据@mazaneicha的建议，我尝试设置更长的超时时间，但仍然得到相同的错误。

implicit val spark: SparkSession = SparkSession
  .builder
  .master("local[1]")
  .config(new SparkConf().setIfMissing("spark.master", "local[1]")
    .set("spark.eventLog.dir", "file:///tmp/spark-events")
    .set("spark.dynamicAllocation.executorIdleTimeout", "100s")  //spark.dynamicAllocation.executorIdleTimeout
    .set("spark.dynamicAllocation.schedulerBacklogTimeout", "100s")    //spark.dynamicAllocation.schedulerBacklogTimeout
  )
  .getOrCreate()

Answer 1

故障的原因实际上是张贴在日志上的：

2021-12-01 15:05:46,906 WARN  [main] streaming.StreamingQueryManager (Logging.scala:logWarning(69)) - Stopping existing streaming query [id=b13a69d7-5a2f-461e-91a7-a9138c4aa716, runId=9cb31852-d276-42d8-ade6-9839fa97f85c], as a new run is being started.

WHy查询被停止了吗？这是因为在Scala中，我基于集合在循环中创建流查询，同时保持所有查询名和所有检查点名称相同。在使它们成为唯一之后(我刚刚使用了集合中的字符串值)，失败问题就消失了。