django-celery redis内存错误

Question

我使用django+celery和redis作为代理，其中一项任务涉及读取大小约为25MB的大文件并返回结果，通过该结果链接另一个任务来处理结果。

我在这里遇到了这个错误，由于我对redis不熟悉，所以我请求帮助。可能的问题是什么？

[2013-06-23 22:45:41,241: ERROR/MainProcess] Unrecoverable error: MemoryError()
Traceback (most recent call last):
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/__init__.py", line 363, in start
    component.start()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 395, in start
    self.consume_messages()
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/celery/worker/consumer.py", line 480, in consume_messages
    readers[fileno](fileno, event)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 770, in handle_event
    self._callbacks[queue](message)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/virtual/__init__.py", line 479, in _callback
    self.qos.append(message, message.delivery_tag)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/kombu/transport/redis.py", line 117, in append
    dumps([message._raw, EX, RK])) \
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1808, in execute
    return execute(conn, stack, raise_on_error)
  File "/home/property/virtualenv/property_env/lib/python2.6/site-packages/redis/client.py", line 1705, in _execute_transaction
    [args for args, options in cmds]))
MemoryError

不确定它是否提供了任何提示，但检查redis服务器上的maxmemory设置似乎不是问题。

redis 127.0.0.1:6379> config get maxmemory
1) "maxmemory"
2) "3758096384"

Answer 1

看起来内存错误不是在redis端，而是在客户端(celery worker)。

我猜是worker内存不足。

您应该确保celery进程可以将来自redis的结果实际分配到内存中。

如果在执行了几个任务之后发生这种情况，这意味着您没有足够的内存来处理您设置的并发性，或者您正在泄漏内存(将来自redis的对象引用留在了某个地方