芹菜后端连接如何添加重试？

Question

我使用的是celery 5.0.1，并使用CELERY_BACKEND_URL作为redis://:password@redisinstance1:6379/0。它工作得很好，但当有一个Redis实例松散连接时，它会中断任务并返回错误。

Exception: Error while reading from socket: (104, 'Connection reset by peer')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 198, in _read_from_socket
    data = recv(self._sock, socket_read_size)
  File "/usr/local/lib/python3.7/dist-packages/redis/_compat.py", line 72, in recv
    return sock.recv(*args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/celery/app/trace.py", line 477, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 154, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 439, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 855, in _store_result
    current_meta = self._get_task_meta_for(task_id)
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/base.py", line 873, in _get_task_meta_for
    meta = self.get(self.get_key_for_task(task_id))
  File "/usr/local/lib/python3.7/dist-packages/celery/backends/redis.py", line 346, in get
    return self.client.get(key)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 1606, in get
    return self.execute_command('GET', name)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 223, in _read_from_socket
    (ex.args,))
redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')


Celery worker: None
Celery task id: 244b56af-7c96-56cf-a01a-9256cfd98ade
Celery retry attempt: 0
Task args: []
Task kwargs: {'address': 'ipadd', 'uid': 'uid', 'hexID': 'hexID', 'taskID': '244b56af-7c96-56cf-a01a-9256cfd98ade'}

当我运行第二个任务时，它工作得很好，在很短的一段时间内，连接出现了一些故障。

我是否可以设置当celery尝试将结果更新到Redis时，如果它返回错误，它将在2-5秒后重试？

我知道如何在任务中设置重试，但这不会导致任务失败。我的任务运行良好，它返回数据，但celery在更新到后端时失去了连接。

Answer 1

要处理connection timeouts，您可以在您的Celery配置中设置以下内容：

app.conf.broker_transport_options = {
    'retry_policy': {
       'timeout': 5.0
    }
}
app.conf.result_backend_transport_options = {
    'retry_policy': {
       'timeout': 5.0
    }
}

您可能会考虑在配置中使用其他Redis backend settings，例如redis_retry_on_timeout。