从多线程驱动程序启动Apache火花SQL作业。

Question

我想从大约1500个带有Spark的远程Oracle表中提取数据，并且我希望有一个多线程应用程序，它可以在每个线程中获取一个表，或者每个线程可能有10个表，然后启动一个spark作业，从它们各自的表中读取数据。

从官方的火花网站https://spark.apache.org/docs/latest/job-scheduling.html可以清楚地看到，这是可行的.

Spark运行的...cluster管理器为跨应用程序的调度提供了便利。其次，在每个星火应用程序中，如果多个“作业”(火花操作)是由不同线程提交的，则它们可能同时运行。如果您的应用程序是通过网络为请求提供服务，这是很常见的。Spark包括一个公平的调度程序来调度每个SparkContext中的资源。

然而，您可能已经注意到，在这样的post Concurrent job Execution in Spark中，在这个类似的问题上没有被接受的答案，而最受欢迎的答案开始于

这并不是真正的火花精神

1. 每个人都知道这不是火花的“精神”
2. 谁在乎火花的精神是什么？这并不意味着什么

以前有人有过这样的工作吗？你非得做什么特别的事吗？在我浪费大量的工作时间之前，我只想得到一些指点。我真的很感激在这方面的任何帮助！

Answer 1

星火上下文是线程安全的，因此可以从多个线程并行调用它。(我正在制作中)

需要注意的一件事是限制运行的线程数量，因为：

1. 执行器内存在所有线程之间共享，您可以从缓存中获取OOM或不断地交换进出内存。
2. cpu是有限的，所以拥有比核心更多的任务不会有任何改进。

Answer 2

您不需要在一个多线程应用程序中提交作业(尽管我认为您没有理由这样做)。只需将您的工作作为单个过程提交即可。有一个脚本，可以一次提交所有这些作业，并将进程推送到后台，或者以纱线-集群模式提交。您的调度程序( spark、mesos、spark集群)只允许某些作业等待，因为它没有空间让所有调度程序基于内存和/或cpu可用性同时运行。

请注意，只有当您真正使用多个分区处理表时，我才能看到您的方法的好处--而不是像我多次看到的那样，只有一个分区。另外，由于您需要处理那么多表，我不确定您会从中受益多少--如果有的话。根据对表数据的处理方式，只运行多个单线程和无火花作业可能更简单。

也见柯伯特他的便条。

Answer 3

同意@lev，我很长时间都在想它，所以我写了一个简单的小代码来确保它的工作，请注意！！为了控制每个驱动程序的工作人员数量，您需要通过合并来限制dataframe/set。

下面是示例代码：

import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession

object SparkMultiThreadExample extends App{

  val TOTAL_WORKERS = 10
  val NUMBER_OF_WORKERS_PER_DRIVER = 2
  val sparkConf = new SparkConf()
  sparkConf.setMaster(s"local[${TOTAL_WORKERS}]")
  val spark = SparkSession.builder().config(sparkConf).getOrCreate()
  val list1 = (0 until 10).toList
  import spark.implicits._
  list1.par.foreach(t => {
    spark.createDataset(list1).coalesce(NUMBER_OF_WORKERS_PER_DRIVER).foreach(i =>   {
  println(s"${Thread.currentThread()}, Driver thread ${t}: This is inside worker ${i} " )
  Thread.sleep(1000)
  println(s"FINISH ${Thread.currentThread()} Driver thread ${t}: This is inside worker ${i} " )
})
}) }

产出：

Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 0 
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 0 
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 5 
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 5 
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 5 
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 0 
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 0 
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 5 
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 5 
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 0 
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 0 
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 5 
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 0 
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 5 
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 5 
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 6 
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 1 
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 6 
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 1 
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 0 
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 5 
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 5 
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 0 
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 6 
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 0 
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 1 
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 6 
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 2 
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 1 
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 6 
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 2 
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 1 
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 6 
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 6 
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 1 
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 1 
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 7 
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 8 
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 3 
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 8 
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 3 
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 2 
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 7 
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 8 
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 2 
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 3 
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 8 
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 4 
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 9 
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 4 
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 8 
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 3 
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 8 
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 3 
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 9 
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 4 
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 4 
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 5 
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 0 
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 0 
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 5 
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 0 
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 5 
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 0 
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 5 
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 5 
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 0 
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 5 
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 0 
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 0 
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 5 
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 0 
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 5 
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 0 
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 5 
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 5 
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 6 
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 0 
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 1 
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 6 
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 1 
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 1 
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 6 
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 1 
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 6 
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 1 
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 6 
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 6 
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 7 
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 1 
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 2 
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 7 
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 2 
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 2 
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 7 
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 2 
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 7 
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 2 
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 7 
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 7 
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 8 
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 2 
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 3 
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 8 
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 3 
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 3 
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 8 
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 3 
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 8 
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 3 
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 8 
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 8 
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 3 
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 4 
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 9 
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 4