pathos.multiprocessing中的WindowsError

Question

import subprocess
import os
import dill
import pathos.multiprocessing as mp

def main(processes):
    list_of_files = os.listdir(os.getcwd())
    pool = mp.ProcessingPool(processes)
    pool.map(subprocess.call, [['program.exe',fname] 
        for fname in list_of_files])

if __name__ == "__main__":
    processes = 12
    main(processes) 

运行上述命令会出现以下错误：

In [1]: %run batch.py
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
c:\3dns\v3.8\batch.py in <module>()
     26     processes = 12
---> 27     main(processes) 

c:\3dns\v3.8\batch.py in main(cores)
     19
---> 20     pool = mp.ProcessingPool(processes)

C:\Anaconda\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multiprocessing
.pyc in __init__(self, *args, **kwds)
     97
     98         # Create a new server if one isn't already initialized
---> 99         self._serve()
    100         return
    101     __init__.__doc__ = AbstractWorkerPool.__init__.__doc__ + __init__.__
doc__

C:\Anaconda\lib\site-packages\pathos-0.2a1.dev0-py2.7.egg\pathos\multiprocessing
.pyc in _serve(self, nodes)
    108         _pool = __STATE['pool']
    109         if not _pool or nodes != _pool.__nodes:
--> 110             _pool = Pool(nodes)
    111             _pool.__nodes = nodes
    112             __STATE['pool'] = _pool

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\pool.pyc in __init__(self, processes, initializer, initargs)
     70     '''
     71     def __init__(self, processes=None, initializer=None, initargs=()):
---> 72         self._inqueue = SimpleQueue()
     73         self._outqueue = SimpleQueue()
     74         self._taskqueue = Queue.Queue()

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\queue.pyc in __init__(self)
    302
    303     def __init__(self):
--> 304         reader, writer = Pipe(duplex=False)
    305         if sys.platform == 'win32':
    306             state = reader, writer, Lock(), None

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\__init__.pyc in Pipe(duplex)
     95     '''
     96     from processing.connection import Pipe
---> 97     return Pipe(duplex)
     98
     99 def cpuCount():

C:\Anaconda\lib\site-packages\processing-0.52_pathos-py2.7-win-amd64.egg\process
ing\connection.pyc in Pipe(duplex)
    230             win32.PIPE_TYPE_MESSAGE | win32.PIPE_READMODE_MESSAGE |
    231             win32.PIPE_WAIT,
--> 232             1, obsize, ibsize, win32.NMPWAIT_WAIT_FOREVER, win32.NULL
    233             )
    234         h2 = win32.CreateFile(

WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect

这是在Windows7 64位机器上的交互式IPython控制台(Python2.7的Anaconda发行版)。我使用pathos版本0.2a1.dev0和Dill版本0.2.2。每当我创建一个'Pool‘时，这个错误似乎是可重现的。

Answer 1

您是否尝试过直接使用Python multiprocessing库？当我最后一次使用它时，它在Windows上运行得很可靠，尽管目前我没有一台Windows机器来测试特定的行为。

Answer 2

我是pathos的作者。事实上，这是@WillAngley建议的。该错误现在已修复。我已经在Windows8.1，python2.7上进行了验证，并且在第三方的主版本或最新版本中也验证了pathos和相关的依赖项。关键问题是更新multiprocessing的分支。这就完成了，并且在一个名为multiprocess的新包中。