Julia -@按顺序产生计算作业，而不是并行

Question

我正在尝试在Julia (版本)中并行运行函数。1.1.0)使用@spawn宏。

我注意到，使用@spawn时，作业实际上是按顺序执行的(尽管来自不同的工作进程)。使用并行计算作业的pmap函数时不会发生这种情况。

下面是调用(在模块hello_module中)应该执行的函数的main.jl程序的代码：

#### MAIN START ####
# deploy the workers
addprocs(4)
# load modules with multi-core functions
@everywhere include(joinpath(dirname(@__FILE__), "hello_module.jl"))

# number of cores
cpus = nworkers()

# print hello world in parallel
hello_module.parallel_hello_world(cpus)

  [1]: https://docs.julialang.org/en/v1/stdlib/Distributed/#Distributed.pmap

...and以下是该模块的代码：

module hello_module    
using Distributed
using Printf: @printf
using Base

"""Print Hello World on STDOUT"""
function hello_world()
    println("Hello World!")
end

"""Print Hello World in Parallel."""
function parallel_hello_world(threads::Int)

    # create array with as many elements as the threads
    a = [x for x=1:threads]

    #= This would perform the computation in parallel
    wp = WorkerPool(workers())
    c = pmap(hello_world, wp, a, distributed=true)
    =#

    # spawn the jobs
    for t in a
        r = @spawn hello_world()
        # @show r
        s = fetch(r)
    end    
end

end # module end

Answer 1

你需要使用绿色线程来管理你的并行性。在Julia中，它是通过使用@sync和@async宏来实现的。请看下面的最小工作示例：

using Distributed

addprocs(3)
@everywhere using Dates
@everywhere function f()
    println("starting at $(myid()) time $(now()) ")
    sleep(1)
    println("finishing at $(myid()) time $(now()) ")
    return myid()^3
end

function test()
    fs = Dict{Int,Future}()
    @sync for w in workers()
        @async fs[w] = @spawnat w f()
    end
    res = Dict{Int,Int}()
    @sync for w in workers()
        @async res[w] = fetch(fs[w])
    end
    res
end

下面的输出清楚地表明这些函数是并行运行的：

julia> test()
      From worker 3:    starting at 3 time 2019-04-02T01:18:48.411
      From worker 2:    starting at 2 time 2019-04-02T01:18:48.411
      From worker 4:    starting at 4 time 2019-04-02T01:18:48.415
      From worker 2:    finishing at 2 time 2019-04-02T01:18:49.414
      From worker 3:    finishing at 3 time 2019-04-02T01:18:49.414
      From worker 4:    finishing at 4 time 2019-04-02T01:18:49.418
Dict{Int64,Int64} with 3 entries:
  4 => 64
  2 => 8
  3 => 27

编辑：

我建议你管理如何分配你的计算。但是，您也可以使用@spawn。请注意，在下面的场景中，作业被同时分配到工作进程上。

function test(N::Int)
    fs = Dict{Int,Future}()
    @sync for task in 1:N
        @async fs[task] = @spawn f()
    end
    res = Dict{Int,Int}()
    @sync for task in 1:N
        @async res[task] = fetch(fs[task])
    end
    res
end

下面是输出：

julia> test(6)
      From worker 2:    starting at 2 time 2019-04-02T10:03:07.332
      From worker 2:    starting at 2 time 2019-04-02T10:03:07.34
      From worker 3:    starting at 3 time 2019-04-02T10:03:07.332
      From worker 3:    starting at 3 time 2019-04-02T10:03:07.34
      From worker 4:    starting at 4 time 2019-04-02T10:03:07.332
      From worker 4:    starting at 4 time 2019-04-02T10:03:07.34
      From worker 4:    finishin at 4 time 2019-04-02T10:03:08.348
      From worker 2:    finishin at 2 time 2019-04-02T10:03:08.348
      From worker 3:    finishin at 3 time 2019-04-02T10:03:08.348
      From worker 3:    finishin at 3 time 2019-04-02T10:03:08.348
      From worker 4:    finishin at 4 time 2019-04-02T10:03:08.348
      From worker 2:    finishin at 2 time 2019-04-02T10:03:08.348
Dict{Int64,Int64} with 6 entries:
  4 => 8
  2 => 27
  3 => 64
  5 => 27
  6 => 64
  1 => 8