具有持久化启用同步问题的Redis-HA舵图

Question

目前，我们正在使用longhorn存储类在kubernetes( reasons )上运行具有3个副本(https://github.com/helm/charts/tree/master/stable/redis-ha v4.4.4)的支持持久化的redis-ha，由于一些未知的原因，主和从不能同步。这可能会发生在重新启动后30分钟或重新启动后1天，最终我们得到以下错误。

从1错误

redis-cli role
1) "slave"
2) "10.43.6.52"
3) (integer) 6379
4) "connect"
5) (integer) -1

1:S 22 Sep 2020 09:53:22.843 * Master replied to PING, replication can continue... 
1:S 22 Sep 2020 09:53:22.858 * Partial resynchronization not possible (no cached master) 
1:S 22 Sep 2020 09:53:28.189 * Full resync from master: bd87e85aa41950b9844c1bcb29a7870b96b53f79:804594411 
1:S 22 Sep 2020 09:53:33.204 # Opening the temp file needed for MASTER <-> REPLICA synchronization: I/O error 
1:S 22 Sep 2020 09:53:33.905 * Connecting to MASTER 10.43.6.52:6379 
1:S 22 Sep 2020 09:53:33.905 * MASTER <-> REPLICA sync started 
1:S 22 Sep 2020 09:53:33.906 * Non blocking connect for SYNC fired the event. 
1:S 22 Sep 2020 09:53:33.906 * Master replied to PING, replication can continue... 
1:S 22 Sep 2020 09:53:33.907 * Partial resynchronization not possible (no cached master) 
1:S 22 Sep 2020 09:53:36.150 * Full resync from master: bd87e85aa41950b9844c1bcb29a7870b96b53f79:804599892 
1:S 22 Sep 2020 09:53:41.163 # Opening the temp file needed for MASTER <-> REPLICA synchronization: I/O error 
1:S 22 Sep 2020 09:53:41.864 * Connecting to MASTER 10.43.6.52:6379 
1:S 22 Sep 2020 09:53:41.864 * MASTER <-> REPLICA sync started 

从2错误

redis-cli role
1) "slave"
2) "10.43.6.52"
3) (integer) 6379
4) "connected"
5) (integer) 809074465

6049:C 22 Sep 2020 09:55:55.091 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: I/O error 
1:S 22 Sep 2020 09:55:55.188 # Background saving error 
1:S 22 Sep 2020 09:56:01.002 * 1 changes in 30 seconds. Saving... 
1:S 22 Sep 2020 09:56:01.002 * Background saving started by pid 6050 
6050:C 22 Sep 2020 09:56:01.004 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: I/O error 
1:S 22 Sep 2020 09:56:01.102 # Background saving error 
1:S 22 Sep 2020 09:56:07.013 * 1 changes in 30 seconds. Saving... 
1:S 22 Sep 2020 09:56:07.014 * Background saving started by pid 6051 
6051:C 22 Sep 2020 09:56:07.016 # Failed opening the RDB file dump.rdb (in server root dir /data) for saving: I/O error 

主误差

redis-cli role
1) "master"
2) (integer) 809012256
3) 1) 1) "10.43.254.123"
      2) "6379"
      3) "809011980"
   2) 1) "10.43.229.244"
      2) "6379"
      3) "0"

3944:C 22 Sep 2020 09:57:16.102 * RDB: 0 MB of memory used by copy-on-write 
1:M 22 Sep 2020 09:57:16.176 * Background saving terminated with success 
1:M 22 Sep 2020 09:57:16.176 * Starting BGSAVE for SYNC with target: replicas sockets 
1:M 22 Sep 2020 09:57:16.177 * Background RDB transfer started by pid 3945 
1:M 22 Sep 2020 09:57:21.283 # Connection with replica 10.43.229.244:6379 lost. 
1:M 22 Sep 2020 09:57:21.286 # Background transfer error 
1:M 22 Sep 2020 09:57:21.601 * Replica 10.43.229.244:6379 asks for synchronization 
1:M 22 Sep 2020 09:57:21.601 * Full resync requested by replica 10.43.229.244:6379 
1:M 22 Sep 2020 09:57:21.601 * Delay next BGSAVE for diskless SYNC 
1:M 22 Sep 2020 09:57:27.241 * Starting BGSAVE for SYNC with target: replicas sockets 
1:M 22 Sep 2020 09:57:27.243 * Background RDB transfer started by pid 3946 
1:M 22 Sep 2020 09:57:32.254 # Connection with replica 10.43.229.244:6379 lost. 
1:M 22 Sep 2020 09:57:32.266 # Background transfer error 
1:M 22 Sep 2020 09:57:32.563 * Replica 10.43.229.244:6379 asks for synchronization 
1:M 22 Sep 2020 09:57:32.563 * Full resync requested by replica 10.43.229.244:6379 
1:M 22 Sep 2020 09:57:32.563 * Delay next BGSAVE for diskless SYNC 
1:M 22 Sep 2020 09:57:38.304 * Starting BGSAVE for SYNC with target: replicas sockets 
1:M 22 Sep 2020 09:57:38.304 * Background RDB transfer started by pid 3947 
1:M 22 Sep 2020 09:57:43.315 # Connection with replica 10.43.229.244:6379 lost. 
1:M 22 Sep 2020 09:57:43.476 # Background transfer error 
1:M 22 Sep 2020 09:57:43.517 * Replica 10.43.229.244:6379 asks for synchronization 
1:M 22 Sep 2020 09:57:43.517 * Full resync requested by replica 10.43.229.244:6379 
1:M 22 Sep 2020 09:57:43.517 * Delay next BGSAVE for diskless SYNC 
1:M 22 Sep 2020 09:57:47.098 * 1 changes in 30 seconds. Saving... 
1:M 22 Sep 2020 09:57:47.098 * Background saving started by pid 3948 
3948:C 22 Sep 2020 09:57:47.124 * DB saved on disk 
3948:C 22 Sep 2020 09:57:47.124 * RDB: 0 MB of memory used by copy-on-write 
1:M 22 Sep 2020 09:57:47.199 * Background saving terminated with success 
1:M 22 Sep 2020 09:57:47.199 * Starting BGSAVE for SYNC with target: replicas sockets 
1:M 22 Sep 2020 09:57:47.199 * Background RDB transfer started by pid 3949 
3949:C 22 Sep 2020 09:57:47.255 * RDB: 1 MB of memory used by copy-on-write 
1:M 22 Sep 2020 09:57:47.299 * Background RDB transfer terminated with success 
1:M 22 Sep 2020 09:57:47.299 # Slave 10.43.229.244:6379 correctly received the streamed RDB file. 
1:M 22 Sep 2020 09:57:47.299 * Streamed RDB transfer with replica 10.43.229.244:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 
1:M 22 Sep 2020 09:57:52.214 # Connection with replica 10.43.229.244:6379 lost. 
1:M 22 Sep 2020 09:57:52.517 * Replica 10.43.229.244:6379 asks for synchronization 
1:M 22 Sep 2020 09:57:52.517 * Full resync requested by replica 10.43.229.244:6379 
1:M 22 Sep 2020 09:57:52.518 * Delay next BGSAVE for diskless SYNC 
1:M 22 Sep 2020 09:57:58.355 * Starting BGSAVE for SYNC with target: replicas sockets 
1:M 22 Sep 2020 09:57:58.357 * Background RDB transfer started by pid 3950 
1:M 22 Sep 2020 09:58:03.422 # Connection with replica 10.43.229.244:6379 lost. 

红宝石：

dir "/data"
port 6379
maxmemory 0
maxmemory-policy volatile-lru
min-replicas-max-lag 5
min-replicas-to-write 1
rdbchecksum yes
rdbcompression yes
repl-diskless-sync yes
save 30 1
timeout 1000
slaveof 10.43.254.123 6379
slave-announce-ip 10.43.6.52
slave-announce-port 6379

到目前为止，我所想到的是：

• 密钥来自于RabbitMQ，有时开发人员关闭用户来堆叠网格，堆叠的消息会对redis造成很大的负载，没有发现任何日志
• Longhorn存储类可能被破坏，没有找到任何日志

。

我愿意听取任何建议。

Answer 1

关于这方面的最新消息，这是由于长角星SC。迁移到SSD都很好。