Pathos池未运行

Question

我在使用多处理的同时获得了PicklingError: Can't pickle <type 'obj'>，所以我从多处理切换到了pathos。

我的代码如下所示：

from pathos.multiprocessing import ProcessingPool as Pool
pool = Pool(processes=9)
tuples = [(session, query, parse_query, filter_values, i) for i in range(32)]

responses = pool.uimap(execute_queries, [session]*32, [query]*32,[parse_query]*32, [filter_values]*32, [i for i in range(32)]  )
pool.close() 
pool.join()
                
response = reduce(reduce_dic, responses)

在运行代码时，我得到了以下错误：Pool not running在一篇文章中，我读到我需要清除池，但我不确定这对我来说是如何工作的。

Answer 1

我是pathos的作者。我假设您在一个封闭的池中运行。在这种情况下，您需要做的就是重新启动池。

>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> p = Pool()
>>> responses = p.uimap(lambda x:x*x, range(32))
>>> p.close()
>>> p.join()
>>> x = list(responses)
>>> x
[0, 1, 16, 4, 9, 36, 25, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]
>>> 
>>> responses = p.uimap(lambda x:x*x, range(32))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/pathos-0.2.9.dev0-py3.7.egg/pathos/multiprocessing.py", line 149, in uimap
    return _pool.imap_unordered(star(f), zip(*args)) # chunksize
  File "/Users/mmckerns/lib/python3.7/site-packages/multiprocess-0.70.13.dev0-py3.7.egg/multiprocess/pool.py", line 332, in imap_unordered
    raise ValueError("Pool not running")
ValueError: Pool not running
>>>
>>>
>>> p.restart()
<multiprocess.pool.Pool object at 0x105031c50>
>>> responses = p.uimap(lambda x:x*x, range(32))
>>> list(responses)
[0, 1, 4, 16, 9, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841, 900, 961]
>>>