python多处理PicklingError

Question

我正在尝试做一个多用途的paraloop类，以便能够轻松地运行多处理器作业。基本上，用户必须将每个迭代步骤定义为with语句中的def iteration(index)，如本文中的示例所示

下面是我的实现

import multiprocessing as mp
import types


class paraloop(object):
    def __init__(self, ncores, niterations):
        self.niterations = niterations
        self.ncores      = min(ncores, self.niterations)

    def __enter__(self, *args, **kwargs):
        self.pool     = mp.Pool(processes = self.ncores)
        self.iterated = 0
        # create results dict
        self.result = {}
        return self

    def __exit__(self, *args, **kwargs):
        print isinstance (iteration, types.MethodType)

        def ITER():
            self.iterated += 1
            self.result[self.iterated] = iteration(self.iterated)
            if self.iterated < self.niterations:
                self.pool.apply_async( ITER ).get()

        print isinstance (ITER, types.MethodType)
        # run iterations in parallel    
        [self.pool.apply_async( ITER ).get() for idx in xrange(self.ncores)]

# usage example
import numpy as np
ITERATIONS = 10
ARRAY = np.ones(1000000)

with paraloop(ncores=4, niterations=ITERATIONS) as p:
    def iteration(index):
        print 'this is an iteration %i'%index
        s = 0
        for n in ARRAY:
            s += n
        return s

print语句是为了确保我有picklable函数，而不是方法。

>> False
>> False
>> Traceback (most recent call last):
>>   File "paraloop.py", line 48, in <module>
>>     def iteration(index):
>>   File "paraloop.py", line 29, in __exit__
>>     [self.pool.apply_async( ITER ).get() for idx in xrange(self.ncores)]
>>   File "c:\Python27\lib\multiprocessing\pool.py", line 558, in get
>>     raise self._value
>> cPickle.PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

你知道为什么我会收到这个错误消息吗？

Answer 1

作为一个任意规则，请考虑到，如果您不能导入它，那么您就不能对其进行pickle处理，因为pickle会在序列化之前尝试这样做。只是不要在另一个函数中定义目标函数！