在用HIGHEST_PROTOCOL实例化过程中从泡菜文件加载类实例

Question

目标

如果存在泡菜文件，则创建一个scratch：

• 实例时，如果不存在泡菜文件，则从泡菜文件
• 加载实例，然后从创建实例

HIGHEST_PROTOCOL.

• 使用泡菜

失败

有人在评论中提到了同样的问题。in this post也描述了这个问题:如果协议> 1，那么泡菜会在加载过程中调用__new__，这会产生无限的递归。

# module_a.py
import os
import pickle
# import dill as pickle

save_path = r'C:\tests\pickle_tests\saved_instance_of_a.pkl'

def load(path):
    with open(path, 'rb') as f:
        return pickle.load(f)

def dump(x, path):
    with open(path, 'wb') as f:
        pickle.dump(
            x, f,
            protocol=pickle.HIGHEST_PROTOCOL)

class ClassA:
    def __new__(cls):
        print('__new__ called')

        if os.path.isfile(save_path):
            print('The saved pickle exists: loading from file.')
            instance = load(save_path)

        else:
            print('The saved pickle does not exist: creating.')
            instance = super(ClassA, cls).__new__(cls)

        return instance

    def __init__(self):
        print('__init__ called')
        if not os.path.isfile(save_path):
            self.my_dict = {'pi': 3.14}
            dump(self, save_path)

# myprogram.py
import os
import module_a

if __name__ =='__main__':
    instance_a = module_a.ClassA()
    print(instance_a.my_dict)

首先运行OK (从头创建实例)：

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}

第二次运行失败(从泡菜加载实例)：

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
The saved pickle exists: loading from file.
__new__ called
...
  File "C:\tests\pickle_tests\module_a.py", line 19, in __new__
    print('__new__ called')
  File "C:\anaconda\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
RecursionError: maximum recursion depth exceeded while calling a Python object

当前的解决办法

如果我用protocol=pickle.HIGHEST_PROTOCOL代替protocol=0，那么上面的代码可以工作，但我不想使用协议0(或1)。我想使用protocol=pickle.HIGHEST_PROTOCOL。

首先运行OK：

$ python myprogram.py
__new__ called
The saved pickle does not exist: creating.
__init__ called
{'pi': 3.14}

第二次运行OK：

$ python myprogram.py
__new__ called
The saved pickle exists: loading from file.
__init__ called
{'pi': 3.14}

Answer 1

由于泡菜调用__new__是导致问题的原因，允许使用HIGHEST_PROTOCOL的简单解决方法是让您自己使用而不是，并在__init__()方法中完成所有操作。

有一种方法可以做到：

myprogram.py

import os
import module_a

if __name__ =='__main__':
    instance_a = module_a.ClassA()
    print(instance_a.my_dict)

module_a.py

import os
import pickle

SAVE_PATH = r'C:\tests\pickle_tests\saved_instance_of_a.pkl'

def load(path):
    with open(path, 'rb') as f:
        return pickle.load(f)

def dump(x, path):
    with open(path, 'wb') as f:
        pickle.dump(x, f, protocol=pickle.HIGHEST_PROTOCOL)


class ClassA:
    def __init__(self):
        print('__init__ called')
        if os.path.isfile(SAVE_PATH):
            print('  loading from pickle file.')
            self.__dict__ = load(SAVE_PATH)
        else:
            print('  creating from scratch.')
            self.my_dict = {'pi': 3.14}
            dump(self.__dict__, SAVE_PATH)