在所有池中使用的ceph大于之和(ceph detail)

Question

首先，对不起，我在ceph集群中的英语很差，当我运行ceph df detail命令时，它显示的结果如下所示

RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED 
    hdd        62 TiB      52 TiB      10 TiB       10 TiB         16.47 
    ssd       8.7 TiB     8.4 TiB     370 GiB      377 GiB          4.22 
    TOTAL      71 TiB      60 TiB      11 TiB       11 TiB         14.96 
 
POOLS:
    POOL                ID     STORED      OBJECTS     USED        %USED     MAX AVAIL     QUOTA OBJECTS     QUOTA BYTES     DIRTY       USED COMPR     UNDER COMPR 
    rbd-kubernetes      36     288 GiB      71.56k     865 GiB      1.73        16 TiB     N/A               N/A              71.56k            0 B             0 B 
    rbd-cache           41     2.4 GiB     208.09k     7.2 GiB      0.09       2.6 TiB     N/A               N/A             205.39k            0 B             0 B 
    cephfs-metadata     51     529 MiB         221     1.6 GiB         0        16 TiB     N/A               N/A                 221            0 B             0 B 
    cephfs-data         52     1.0 GiB         424     3.1 GiB         0        16 TiB     N/A               N/A                 424            0 B             0 B 

所以我有一个关于结果的问题，正如你所看到的，我使用的存储池的总和小于1TB，但是在RAW STORAGE部分，来自HDD硬盘的使用量是10 TB，并且每天都在增长。我认为这是不寻常的，这个CEPH集群有问题。

同时，ceph osd dump | grep replicated的输出是

pool 36 'rbd-kubernetes' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 244 pg_num_target 64 pgp_num_target 64 last_change 1376476 lfor 2193/2193/2193 flags hashpspool,selfmanaged_snaps,creating tiers 41 read_tier 41 write_tier 41 stripe_width 0 application rbd
pool 41 'rbd-cache' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 1376476 lfor 2193/2193/2193 flags hashpspool,incomplete_clones,selfmanaged_snaps,creating tier_of 36 cache_mode writeback target_bytes 1000000000000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 3600s x1 decay_rate 0 search_last_n 0 min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0
pool 51 'cephfs-metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 31675 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
pool 52 'cephfs-data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 742334 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs

ceph -v版

ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)

ceph tell osd.* version版本返回所有OSD，如

osd.0: {
    "version": "ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable)"
}

Ceph ceph -s

  cluster:
    id:     6a86aee0-3171-4824-98f3-2b5761b09feb
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum ceph-sn-03,ceph-sn-02,ceph-sn-01 (age 37h)
    mgr: ceph-sn-01(active, since 4d), standbys: ceph-sn-03, ceph-sn-02
    mds: cephfs-shared:1 {0=ceph-sn-02=up:active} 2 up:standby
    osd: 63 osds: 63 up (since 41h), 63 in (since 41h)
 
  task status:
    scrub status:
        mds.ceph-sn-02: idle
 
  data:
    pools:   4 pools, 384 pgs
    objects: 280.29k objects, 293 GiB
    usage:   11 TiB used, 60 TiB / 71 TiB avail
    pgs:     384 active+clean

Answer 1

根据所提供的数据，您应该评估以下考虑事项和方案：

1. 复制大小是包含的，一旦在写操作中实现了min_size，就会收到一条完成消息。这意味着您应该期望存储消耗最小的min_size和最大的复制大小.

。

1. ceph存储元数据和日志用于内务管理，显然需要消耗存储。

如果通过"rados“或与参数类似的接口进行基准测试操作，则对象将永久存储在消耗存储的集群中.

所有提到的场景都有几种可能性。