Ceph群集-降低数据可用性: 96 pgs不活动，所有OSD节点都已关闭

Question

我按照这文档设置了Ceph集群。我有一个Manager节点、一个Monitor节点和三个OSD节点。问题是，在我设置完集群之后，ceph health返回了所有三个节点的HEALTH_OK。然而，当我回到我的集群时，情况就不太好了。这是健康检查的输出：

HEALTH_WARN Reduced data availability: 96 pgs inactive
PG_AVAILABILITY Reduced data availability: 96 pgs inactive
    pg 0.0 is stuck inactive for 35164.889973, current state unknown, last acting []
    pg 0.1 is stuck inactive for 35164.889973, current state unknown, last acting []
    pg 0.2 is stuck inactive for 35164.889973, current state unknown, last acting []

对其他的朋友也是一样。我不知道为什么会发生这种事。我使用的是Ceph13.2.10模拟版(稳定版)。我已经寻找了一个答案，但其他似乎有相同问题的人并没有遇到节点故障。我所有的osd节点都关闭了，这是ceph -s的输出：

  cluster:
    id:     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx
    health: HEALTH_WARN
            Reduced data availability: 96 pgs inactive

  services:
    mon: 1 daemons, quorum server-1
    mgr: server-1(active)
    osd: 3 osds: 0 up, 0 in

  data:
    pools:   2 pools, 96 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             96 unknown

我还检查了osd日志，我不明白问题是什么，但这几行表示我的Ceph版本有问题，我必须升级到发光版，但我已经有了一个更新的版本：

2021-02-18 22:01:11.994 7fb070e25c00  0 osd.1 14 done with init, starting boot process
2021-02-18 22:01:11.994 7fb070e25c00  1 osd.1 14 start_boot
2021-02-18 22:01:11.998 7fb049add700 -1 osd.1 14 osdmap require_osd_release < luminous; please upgrade to luminous
2021-02-18 22:11:00.706 7fb050aeb700 -1 osd.1 15 osdmap require_osd_release < luminous; please upgrade to luminous
2021-02-18 22:35:52.276 7fb050aeb700 -1 osd.1 16 osdmap require_osd_release < luminous; please upgrade to luminous
2021-02-18 22:36:08.836 7fb050aeb700 -1 osd.1 17 osdmap require_osd_release < luminous; please upgrade to luminous
2021-02-19 04:05:00.895 7fb0512ec700  1 bluestore(/var/lib/ceph/osd/ceph-1) _balance_bluefs_freespace gifting 0x1f00000~100000 to bluefs
2021-02-19 04:05:00.931 7fb0512ec700  1 bluefs add_block_extent bdev 1 0x1f00000~100000
2021-02-19 04:23:51.208 7fb0512ec700  1 bluestore(/var/lib/ceph/osd/ceph-1) _balance_bluefs_freespace gifting 0x2400000~400000 to bluefs
2021-02-19 04:23:51.244 7fb0512ec700  1 bluefs add_block_extent bdev 1 0x2400000~400000

我还通过ceph tell osd.* version检查了osd版本，这是输出：

Error ENXIO: problem getting command descriptions from osd.0
osd.0: problem getting command descriptions from osd.0
Error ENXIO: problem getting command descriptions from osd.1
osd.1: problem getting command descriptions from osd.1
Error ENXIO: problem getting command descriptions from osd.2
osd.2: problem getting command descriptions from osd.2

而ceph-osd --version则返回Ceph13.2.10版本的模拟(稳定)。

我不明白这会是什么问题。我也尝试过systemctl start -l ceph-osd@#，但没有成功。我不知道我还能尝试什么，也不知道为什么会发生这样的事情。

Answer 1

我记得我经历过几次同样的问题。一旦问题是iptables，我忘了在监视器和OSD上为集群网络打开端口。另一次是因为我的crushmap失败域被设置为主机，并且我在一个集群中运行了一个所有的程序，通过将crushmap设置为osd解决了问题。