Elasticsearch jdbc river耗尽了整个内存

Question

我正在尝试将mysql表中的1600万个文档(47 to )索引到elasticsearch索引中。我正在使用jparante's elasticsearch jdbc river来做这件事。但是，在创建了河并等待了大约15分钟后，整个堆内存都被消耗了，没有任何河流运行的迹象或文档被索引。当我有大约1000-1200万条记录要索引时，这条河过去运行得很好。我试着在这条河上跑了3-4次，但都是徒劳的。

Heap Memory pre allocated to the ES process = 10g

elasticsearch.yml

 cluster.name: test_cluster

 index.cache.field.type: soft
 index.cache.field.max_size: 50000
 index.cache.field.expire: 2h

 cloud.aws.access_key: BBNYJC25Dij8JO7YM23I(fake)
 cloud.aws.secret_key: GqE6y009ZnkO/+D1KKzd6M5Mrl9/tIN2zc/acEzY(fake)
 cloud.aws.region: us-west-1

 discovery.type: ec2
 discovery.ec2.groups: sg-s3s3c2fc(fake)
 discovery.ec2.any_group: false
 discovery.zen.ping.timeout: 3m

 gateway.recover_after_nodes: 1
 gateway.recover_after_time: 1m

 bootstrap.mlockall: true

 network.host: 10.111.222.33(fake)

river.sh

curl -XPUT 'http://--address--:9200/_river/myriver/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "driver" : "com.mysql.jdbc.Driver",
        "url" : "jdbc:mysql://--address--:3306/mydatabase",
        "user" : "USER",
        "password" : "PASSWORD",
        "sql" : "select * from mytable order by creation_time desc",
        "poll" : "5d",
        "versioning" : false
    },
    "index" : {
        "index" : "myindex",
        "type" : "mytype",
        "bulk_size" : 500,
        "bulk_timeout" : "240s"
    }
}'

系统属性：

16gb RAM
200gb disk space

Answer 1

根据您的elasticsearch-river-jdbc版本(使用ls -lrt plugins/river-jdbc/查找)，此错误可能会被关闭(https://github.com/jprante/elasticsearch-river-jdbc/issues/45)

否则，请在Github上提交错误报告。