大容量插入的弹性搜索内存错误

Question

我一次插入5000条记录到弹性搜索中，这些记录的总大小是: 33936 (我是用sys.getsizeof()获得的)

弹性搜索版本: 1.5.0 Python2.7 Ubuntu

以下是以下错误：

Traceback (most recent call last):
  File "run_indexing.py", line 67, in <module>
    index_policy_content(datatable, source, policyids)
  File "run_indexing.py", line 60, in index_policy_content
    bulk(elasticsearch_instance, actions)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers.py", line 148, in bulk
    for ok, item in streaming_bulk(client, actions, **kwargs):
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers.py", line 107, in streaming_bulk
    resp = client.bulk(bulk_actions, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 70, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 568, in bulk
    params=params, body=self._bulk_body(body))
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 259, in perform_request
    body = body.encode('utf-8')
MemoryError

请帮我解决这个问题。

谢谢和问候，非洲裔

Answer 1

如果我不得不猜测，我会说这个内存错误发生在python中，因为它加载并序列化了它的数据。试着减少批处理的大小，直到得到有用的东西，然后向上搜索直到它再次失败。这应该可以帮助你找到一个安全的批次大小来使用。

(其他有用的信息可能包括:运行python进程的服务器中的内存量、elasticsearch服务器节点的内存量、分配给Java的堆数量。)