无法从spyder IDE运行pathos程序

Question

我有以下简单的程序：

from pathos.core import connect

tunnel = connect('192.168.1.5', port=50004)

print(tunnel)
print(type(tunnel._lport))
print(tunnel._rport)

def sleepy_squared(x):
    from time import sleep
    sleep(1.0)
    return x**2

from pathos.pp import ParallelPythonPool as Pool
p = Pool(8, servers=('192.168.1.5:6260',))
print(p.servers)

x = [1, 2, 3, 4, 5, 6, 7, 8, 9]

y = p.map(sleepy_squared, x)
print(y)

当我尝试从Spyder 4 IDE运行这个程序时，我得到以下错误：

Tunnel('-q -N -L 4761:192.168.1.5:50004 192.168.1.5')
<class 'int'>
50004
('192.168.1.5:6260',)
Traceback (most recent call last):

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3319, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-1-e89974d31563>", line 20, in <module>
y = p.map(sleepy_squared, x)

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 234, in map
return list(self.imap(f, *args))
  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 247, in imap
return (subproc() for subproc in list(builtins.map(submit, *args)))

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 243, in submit
return _pool.submit(f, argz, globals=globals())

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 499, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 683, in __dumpsfunc
sources = [self.__get_source(func) for func in funcs]

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 683, in <listcomp>
sources = [self.__get_source(func) for func in funcs]

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 750, in __get_source
self.__sourcesHM[hashf] = importable(func)

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 957, in importable
src = _closuredimport(obj, alias=alias, builtin=builtin)
  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 876, in _closuredimport
src = getimport(func, alias=alias, builtin=builtin)

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 764, in getimport
return _getimport(head, tail, alias, verify, builtin)

  File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 713, in _getimport
try: exec(_str) #XXX: check if == obj? (name collision)

  File "<string>", line 1
from __main__'> import sleepy_squared
                                     ^
SyntaxError: EOL while scanning string literal

当我使用下面的命令python test_connect.py从终端运行这个程序时，程序运行得很好。我的问题是，为什么程序不能在Spyder IDE 4上运行，我如何才能使程序在Spyder IDE 4上运行？

Answer 1

我是pathos的作者。Jupyter和其他IDE在解释器之上添加了一个额外的执行层，在某些情况下，甚至将执行封装在一个闭包中，以将额外的钩子添加到IDE的其余部分。您正在使用一个ParallelPool，它使用ppft，它使用dill.source通过提取对象的源代码及其依赖项来“序列化”。由于集成开发环境添加了一个闭合层，Spyder也必须尝试序列化它，但这并不成功--所以简而言之，这是dill.source和dill.source之间的兼容性问题。如果您选择其他pathos池之一，则可能会成功。ProcessPool在本质上与ParallelPool相同，但它是按对象而不是按源代码序列化的--它使用multiprocess，后者使用dill。然后是ThreadPool，它可能是最有可能成功的，除非Spyder也扰乱主线程--大多数IDE都会这样做。那么，你能做些什么呢？简单的事情是不从IDE运行并行代码。本质上，在集成开发环境中编写代码，然后换出Pool，它就应该并行运行。IDE通常不能很好地处理并行计算。