火花流应用中的Spark透镜性能分析

Question

我正在尝试使用火花流应用程序进行性能分析，该应用程序使用的是S以来镜头。它给出了这样的结果

Executor count     1  ( 80%) estimated time 01m 29s and estimated cluster utilization 18.55%
Executor count     2  (100%) estimated time 01m 28s and estimated cluster utilization 9.36%
Executor count     3  (150%) estimated time 01m 28s and estimated cluster utilization 6.26%
Executor count     4  (200%) estimated time 01m 28s and estimated cluster utilization 4.69%
Executor count     6  (300%) estimated time 01m 28s and estimated cluster utilization 3.13%
Executor count     8  (400%) estimated time 01m 28s and estimated cluster utilization 2.35%
Executor count    10  (500%) estimated time 01m 28s and estimated cluster utilization 1.88%

即使我增加遗嘱执行人，估计的时间也是一样的。

这些建议是否正确？

Answer 1

你为什么不试试呢？很可能您正在运行单个任务。单个任务将不会受益于额外的核心，除非您找到了将其分解为并行任务的方法。