如何去除RelStorage包中的气泡？

Question

这个问题与如何用Plone和RelStorage包装blobstorage有关

使用以佐达布和sqlite为后端的RelStorage数据库，我正在尝试删除未使用的blobs。目前，db.pack不从光盘中删除气泡。下面的最低工作示例演示了此行为：

import logging
import numpy as np
import os
import persistent
from persistent.list import PersistentList
import shutil
import time
from ZODB import config, blob

connectionString = """
%import relstorage
<zodb main>
<relstorage>
blob-dir ./blob
keep-history false
cache-local-mb 0
<sqlite3>
    data-dir .
</sqlite3>
</relstorage>
</zodb>
"""


class Data(persistent.Persistent):
    def __init__(self, data):
        super().__init__()

        self.children = PersistentList()

        self.data = blob.Blob()
        with self.data.open("w") as f:
            np.save(f, data)


def main():
    logging.basicConfig(level=logging.INFO)
    # Initial cleanup
    for f in os.listdir("."):
        if f.endswith("sqlite3"):
            os.remove(f)

    if os.path.exists("blob"):
        shutil.rmtree("blob", True)

    # Initializing database
    db = config.databaseFromString(connectionString)
    with db.transaction() as conn:
        root = Data(np.arange(10))
        conn.root.Root = root

        child = Data(np.arange(10))
        root.children.append(child)

    # Removing child reference from root
    with db.transaction() as conn:
        conn.root.Root.children.pop()

    db.close()

    print("blob directory:", [[os.path.join(rootDir, f) for f in files] for rootDir, _, files in os.walk("blob") if files])
    db = config.databaseFromString(connectionString)
    db.pack(time.time() + 1)
    db.close()
    print("blob directory:", [[os.path.join(rootDir, f) for f in files] for rootDir, _, files in os.walk("blob") if files])


if __name__ == "__main__":
    main()

上面的例子做了以下工作：

1. 删除当前目录中的任何先前数据库以及blob目录。
2. 从头创建数据库/存储，添加两个对象(根和子对象)，根引用子对象并执行事务。
3. 删除根到子的链接并执行事务。
4. 关闭数据库/存储
5. 打开数据库/存储，并在以后执行db.pack一秒钟。

最小工作示例的输出如下：

INFO:ZODB.blob:(23376) Blob directory '<some path>/blob/' does not exist. Created new directory.
INFO:ZODB.blob:(23376) Blob temporary directory './blob/tmp' does not exist. Created new directory.
blob directory: [['blob/.layout'], ['blob/3/.lock', 'blob/3/0.03da352c4c5d8877.blob'], ['blob/6/.lock', 'blob/6/0.03da352c4c5d8877.blob']]
INFO:relstorage.storage.pack:pack: beginning pre-pack
INFO:relstorage.storage.pack:Analyzing transactions committed Thu Aug 27 11:48:17 2020 or before (TID 277592791412927078)
INFO:relstorage.adapters.packundo:pre_pack: filling the pack_object table
INFO:relstorage.adapters.packundo:pre_pack: Filled the pack_object table
INFO:relstorage.adapters.packundo:pre_pack: analyzing references from 7 object(s) (memory delta: 256.00 KB)
INFO:relstorage.adapters.packundo:pre_pack: objects analyzed: 7/7
INFO:relstorage.adapters.packundo:pre_pack: downloading pack_object and object_ref.
INFO:relstorage.adapters.packundo:pre_pack: traversing the object graph to find reachable objects.
INFO:relstorage.adapters.packundo:pre_pack: marking objects reachable: 4
INFO:relstorage.adapters.packundo:pre_pack: finished successfully
INFO:relstorage.storage.pack:pack: pre-pack complete
INFO:relstorage.adapters.packundo:pack: will remove 3 object(s)
INFO:relstorage.adapters.packundo:pack: cleaning up
INFO:relstorage.adapters.packundo:pack: finished successfully
blob directory: [['blob/.layout'], ['blob/3/.lock', 'blob/3/0.03da352c4c5d8877.blob'], ['blob/6/.lock', 'blob/6/0.03da352c4c5d8877.blob']]

正如您所看到的，db.pack确实删除了3个对象“将删除3个对象”，但是文件系统中的blobs没有改变。

在RelStorage的单元测试中，它们似乎会测试是否从文件系统(请看这里)中删除了blobs，但是在上面的脚本中它不起作用。

我做错了什么？任何提示/链接/帮助都将不胜感激。

Answer 1

默认情况下，blob存储目录用作缓存，存储也存储在数据库中的blob数据；其思想是，从本地磁盘缓存加载blob数据比从远程数据库服务器加载blob数据要快。在没有历史记录的存储中使用缓存blob存储不会删除不可访问的blob文件，而是依赖文件大小限制器在需要腾出空间时删除陈旧的缓存数据。但是，您没有设置大小限制，因此缓存变得无限，那些不可访问的blob文件将永远存在。

打包不能在这里删除blob文件，因为缓存是每个ZODB客户端的本地缓存；它不属于ZODB存储的管辖范围。当使用SQLite作为数据库层时，这可能不那么明显，但是想象一下，在一个单独的服务器上使用Postgres，在不同的计算机上有多个客户机，您可以看到打包时缓存清理是不可行的。

注意，另一个blob存储选项是共享blob存储，它可能更接近您的预期:所有blob数据都存储在磁盘上，而不是存储在数据库中。当与远程数据库服务器和多个客户端一起使用时，您需要将其放在类似NTFS共享的东西上。在这种情况下，包装直接操作于blob，当您打包时，无法访问的blob文件立即被移除。

你有两个选择：

• 通过设置blob-cache-size设置blob缓存的大小限制。打包仍然不会删除blob文件，但是当空间不足时，它们将被删除。
• 切换到共享blob缓存(将shared-blob-dir设置为true)。对于以spite为背景的RelStorage来说，这可能比缓存blob存储更有意义，尽管文档中有可怕的警告！

因此，最简单的更改是切换blob存储模式：

connectionString = """
%import relstorage
<zodb main>
<relstorage>
blob-dir ./blob
shared-blob-dir true
keep-history false
cache-local-mb 0
<sqlite3>
    data-dir .
</sqlite3>
</relstorage>
</zodb>
"""

然后，输出更改为：

INFO:ZODB.blob:(26177) Blob directory '<some path>/blob/' does not exist. Created new directory.
INFO:ZODB.blob:(26177) Blob temporary directory './blob/tmp' does not exist. Created new directory.
blob directory: [['blob/.layout'], ['blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x03/0x03da4f169582cd22.blob', 'blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x03/.lock'], ['blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x06/0x03da4f169582cd22.blob', 'blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x06/.lock']]
INFO:relstorage.storage.pack:pack: beginning pre-pack
INFO:relstorage.storage.pack:Analyzing transactions committed Tue Sep  1 01:22:35 2020 or before (TID 277621285453417864)
INFO:relstorage.adapters.packundo:pre_pack: filling the pack_object table
INFO:relstorage.adapters.packundo:pre_pack: Filled the pack_object table
INFO:relstorage.adapters.packundo:pre_pack: analyzing references from 7 object(s) (memory delta: 0 KB)
INFO:relstorage.adapters.packundo:pre_pack: objects analyzed: 7/7
INFO:relstorage.adapters.packundo:pre_pack: downloading pack_object and object_ref.
INFO:relstorage.adapters.packundo:pre_pack: traversing the object graph to find reachable objects.
INFO:relstorage.adapters.packundo:pre_pack: marking objects reachable: 4
INFO:relstorage.adapters.packundo:pre_pack: finished successfully
INFO:relstorage.storage.pack:pack: pre-pack complete
INFO:relstorage.adapters.packundo:pack: will remove 3 object(s)
INFO:relstorage.adapters.packundo:pack: cleaning up
INFO:relstorage.adapters.packundo:pack: finished successfully
blob directory: [['blob/.layout'], ['blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x03/0x03da4f169582cd22.blob', 'blob/0x00/0x00/0x00/0x00/0x00/0x00/0x00/0x03/.lock']]

是的，blob dir布局改变了，所以它可以处理所有可能的OID，永远。然而，OID 6已被删除。

您发现的单元测试只有在使用共享blob缓存进行测试时才能运行。

# If the blob directory is a cache, don't test packing,
# since packing can not remove blobs from all caches.
test_packing = shared_blob_dir