弹性搜索主灾难恢复

Question

我们有一个弹性搜索簇，包含5个数据节点和2个主节点。始终禁用一个主节点上的弹性搜索服务，因此始终只有一个主节点处于活动状态。今天，由于某种原因，当前的主节点出现故障。我们在第二个主节点上启动了服务。所有连接到新主节点的数据节点，都成功地分配了所有主碎片，但是所有副本都没有被分配，我留下了几乎384个未分配的碎片。

我现在该怎么做，来分配他们？

在这种情况下必须采取的最佳做法和步骤是什么？

下面是我的settings的样子：http://pastebin.com/mK1QBfP6

当我尝试手动分配碎片时，我会得到以下错误：

➜  Desktop curl -XPOST http://localhost:9200/_cluster/reroute\?pretty -d '{
  "commands": [
    {
      "allocate": {
        "index": "logstash-1970.01.18",
        "shard": 1,
        "node": "node-name",
        "allow_primary": true
      }
    }
  ]
}'
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [logstash-1970.01.18][1] on node {node-name}{vrVG4CBbSvubWHOzn2qfQA}{10.100.0.146}{10.100.0.146:9300}{master=false} is not allowed, reason: [YES(allocation disabling is ignored)][NO(more than allowed [85.0%] used disk on node, free: [13.671127301258165%])][YES(shard not primary or relocation disabled)][YES(target node version [2.2.0] is same or newer than source node version [2.2.0])][YES(no allocation awareness enabled)][YES(shard is not allocated to same node or host)][YES(allocation disabling is ignored)][YES(below shard recovery limit of [2])][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(node passes include/exclude/require filters)][YES(primary is already active)]"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "[allocate] allocation of [logstash-1970.01.18][1] on node {node-name}{vrVG4CBbSvubWHOzn2qfQA}{10.100.0.146}{10.100.0.146:9300}{master=false} is not allowed, reason: [YES(allocation disabling is ignored)][NO(more than allowed [85.0%] used disk on node, free: [13.671127301258165%])][YES(shard not primary or relocation disabled)][YES(target node version [2.2.0] is same or newer than source node version [2.2.0])][YES(no allocation awareness enabled)][YES(shard is not allocated to same node or host)][YES(allocation disabling is ignored)][YES(below shard recovery limit of [2])][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(node passes include/exclude/require filters)][YES(primary is already active)]"
  },
  "status" : 400
}

任何帮助都将不胜感激。

Answer 1

因此，下面是我为分配未分配的碎片所做的事情：

生成5个新的ES-数据服务器，并等待它们加入集群。一旦它们在集群中，我使用了以下脚本：

#!/bin/bash
array=(node1 node2 node3 node4 node5)
node_counter=0
length=${#array[@]}
IFS=$'\n'
for line in $(curl -s 'http://ip-adress:9200/_cat/shards'|  fgrep UNASSIGNED); do
    INDEX=$(echo $line | (awk '{print $1}'))
    SHARD=$(echo $line | (awk '{print $2}'))
    NODE=${array[$node_counter]}
    echo $NODE
    curl -XPOST 'http://IP-adress:9200/_cluster/reroute' -d '{
        "commands": [
        {
            "allocate": {
                "index": "'$INDEX'",
                "shard": '$SHARD',
                "node": "'$NODE'",
                "allow_primary": true
            }
        }
        ]
    }'
    node_counter=$(((node_counter)%length +1))
done

若要将未分配的碎片分配给新数据节点，请执行以下操作。集群需要5比6的时间才能再次恢复。虽然这是黑客行为，但一个相关的答案会更有意义。

以下是未回答的问题：

• 碎片已经在老的节点上了，为什么ES-Master没有意识到这一点？
• 我们如何显式地要求ES-MASTER扫描已经存在的数据节点并从它们获取信息(关于它们的当前状态、副本、包含的碎片等)