用Django安装redis集群

Question

问题声明使用django-redis与redis聚类。

与缓存交互时会发生错误。错误指向ConnectionPool类实例上的泡菜操作，其中其属性之一线程锁无法序列化，并导致以下错误：

  TypeError: cannot pickle '_thread.lock' object

以复制步骤来再现行为：

1. 在Django设置文件中运行REDIS_URL指向的.
2. 安装CACHES的redis集群。缓存={“默认”：{“后端”："django_redis.cache.RedisCache"，“位置”：REDIS_URL，“选项”：{ "REDIS_CLIENT_CLASS"："redis.cluster.RedisCluster"，"REDIS_CLIENT_KWARGS"：{ "url"：REDIS_URL，}，} }

运行Django控制台并尝试与缓存交互，例如cache.get("somekey")

期望行为来自Redis集群的键的值。

堆栈跟踪

Traceback (most recent call last):
  File "python3.9/site-packages/redis/cluster.py", line 1454, in initialize
    copy_kwargs = copy.deepcopy(kwargs)
  File "python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "python3.9/site-packages/IPython/core/interactiveshell.py", line 3552, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-48456cec0da8>", line 1, in <cell line: 1>
    from django.core.cache import cache; cache.get("bingo")
  File "python3.9/site-packages/django_redis/cache.py", line 91, in get
    value = self._get(key, default, version, client)
  File "python3.9/site-packages/django_redis/cache.py", line 31, in _decorator
    return method(self, *args, **kwargs)
  File "python3.9/site-packages/django_redis/cache.py", line 98, in _get
    return self.client.get(key, default=default, version=version, client=client)
  File "python3.9/site-packages/django_redis/client/default.py", line 253, in get
    client = self.get_client(write=False)
  File "python3.9/site-packages/django_redis/client/default.py", line 105, in get_client
    self._clients[index] = self.connect(index)
  File "python3.9/site-packages/django_redis/client/default.py", line 118, in connect
    return self.connection_factory.connect(self._server[index])
  File "python3.9/site-packages/django_redis/pool.py", line 72, in connect
    connection = self.get_connection(params)
  File "python3.9/site-packages/django_redis/pool.py", line 92, in get_connection
    return self.redis_client_cls(
  File "python3.9/site-packages/redis/cluster.py", line 592, in __init__
    self.nodes_manager = NodesManager(
  File "python3.9/site-packages/redis/cluster.py", line 1286, in __init__
    self.initialize()
  File "python3.9/site-packages/redis/cluster.py", line 1490, in initialize
    raise RedisClusterException(
redis.exceptions.RedisClusterException: ERROR sending "cluster slots" command to redis server 127.0.0.1:7000. error: cannot pickle '_thread.lock' object

Environment:

7.0.0

• redis-py版本:3.3.1

F 238

Answer 1

问题是django_redis.pool.ConnectionFactory.get_connection生成一个池(redis.connection.ConnectionPool类)

  pool = self.get_or_create_connection_pool(params)
  return self.redis_client_cls(
      connection_pool=pool, **self.redis_client_cls_kwargs)

并把它传递给

redis.cluster.RedisCluster.__init__( ... **kwargs ) (so kwargs now has connection_pool)
redis.cluster.NodesManager.__init__
redis.cluster.NodesManager.initialize()
    kwargs = self.connection_kwargs
    copy_kwargs = copy.deepcopy(kwargs)     *******
    *** die!  ConnectionPool has a _thread.lock, not pickle-able. ***

因为redis.connection.ConnectionPool有_thread.lock，所以深度复制()失败。

解决方案#1: 6直接从django_redis.client.DefaultClient.connect()调用django_redis.client.DefaultClient.connect()，绕过ConnectionFactory和ConnectionPool。ConnectionPool.get_connection()制造致命的connection_pool。

解决方案2:创建一个RedisCluster的shim类，将"connection_pool“从kwargs 7中删除。

解决方案3:在使用RedisCluster之前先处理好它。redis.cluster.RedisCluster.__ init __在将它传递给NodesManager之前先擦洗它。

    kwargs = cleanup_kwargs(**kwargs)

它移除redis.cluster.KWARGS_DISABLED_KEYS中列出的键，该键具有

  KWARGS_DISABLED_KEYS = ("host", "port")

8添加"connection_pool“，以便cleanup_kwargs()移除讨厌的键。

参考文献

6参见cKurultayKalkan msg 2022-10-28 django-redis + redis-py 4.0 (与集群一起)向redis.cluster.RedisCluster添加一个shim

class CustomRedisCluster(DefaultClient):
    def connect(self, index):
        """Override the connection retrival function."""
        return RedisCluster.from_url(self._server[index])

在settings.py中，

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": REDIS_URL,
        "OPTIONS": {
            "CLIENT_CLASS": "redis_client.CustomRedisCluster",
        },
    }
}

7在RedisCluster前面添加一个适配器类，以摆脱烦人的“connection_pool”。

import redis.cluster
class RedisClusterShim( redis.cluster.RedisCluster ) :
    def __init__( self,   *args, **kwargs ) :
        kwargs.pop( 'connection_pool', None )
        # redis.cluster.NodesManager.initialize() does connection_pool.deepcopy,
        # but redis.connection.ConnectionPool has a _thread.lock, not pickle-able.
        super().__init__( *args, **kwargs )

8.在使用RedisCluster之前处理它，例如。将其添加到settings.py

def messWithRedisCluster() :
    import redis
    redis.cluster.KWARGS_DISABLED_KEYS = ("host", "port", "connection_pool" )
messWithRedisCluster()