无症状的磁盘问题: dmesg报告和风险评估

Question

我们有一个由10个节点组成的Cassandra集群，其中四个节点显示了dmesg报告的一些磁盘错误。这些报告几乎每小时都会出现。

错误中报告的磁盘是8TB逻辑卷-the的一半，其他磁盘似乎没有报告错误。

[Wed Sep 28 12:29:02 2022] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[Wed Sep 28 12:29:02 2022] sas: ata7: end_device-0:0: cmd error handler
[Wed Sep 28 12:29:02 2022] sas: ata7: end_device-0:0: dev error handler
[Wed Sep 28 12:29:02 2022] ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Wed Sep 28 12:29:02 2022] sas: ata8: end_device-0:1: dev error handler
[Wed Sep 28 12:29:02 2022] ata7.00: failed command: READ DMA EXT
[Wed Sep 28 12:29:02 2022] ata7.00: cmd 25/00:08:10:d8:5d/00:00:84:01:00/e0 tag 8 dma 4096 in
         res 51/40:00:10:d8:5d/00:00:84:01:00/e0 Emask 0x9 (media error)
[Wed Sep 28 12:29:02 2022] ata7.00: status: { DRDY ERR }
[Wed Sep 28 12:29:02 2022] ata7.00: error: { UNC }
[Wed Sep 28 12:29:02 2022] ata7.00: supports DRM functions and may not be fully accessible
[Wed Sep 28 12:29:02 2022] ata7.00: supports DRM functions and may not be fully accessible
[Wed Sep 28 12:29:02 2022] ata7.00: configured for UDMA/133
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] CDB: Read(16) 88 00 00 00 00 01 84 5d d8 10 00 00 00 08 00 00
[Wed Sep 28 12:29:02 2022] blk_update_request: I/O error, dev sda, sector 6515709968
[Wed Sep 28 12:29:02 2022] ata7: EH complete
[Wed Sep 28 12:29:02 2022] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 1 tries: 1
[Wed Sep 28 12:29:02 2022] ata7.00: Enabling discard_zeroes_data

对于R/W比率和平均队列大小，iostat -xdct 1 100显示了非常合理的数字：

grep "^sda" iostat-xdct | awk '{n+=1; r+=$6; w+=$7; qsz+=$9; qmx=qmx<$9?$9:qmx; qmi=qmi>$9?$9:qmi}END{printf("%.3f\t%.3f\t%.3f\t\t[%.3f , %.3f]\n",r/n,w/n,qsz/n,qmi,qmx)}'

818.239 6.939   0.038           [0.000 , 0.890]

根据我们通常的衡量标准，该系统运行良好。然而，我担心这可能会在最意想不到的时候导致致命的系统错误。

我不知道报告的错误意味着什么。我更不知道如何评估让系统像这样运行所涉及的风险。我应该立即更换磁盘吗？

我将粘贴smartctl -a /dev/sda的输出：

smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-693.21.1.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Samsung SSD 870 EVO 4TB
Serial Number:    S6BBNJ0R700321V
LU WWN Device Id: 5 002538 f31706f50
Firmware Version: SVT01B6Q
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   Unknown(0x09fc), ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA >3.2 (0x1ff), 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Sep 28 12:58:46 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (    0) seconds.
Offline data collection
capabilities:            (0x53) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    No Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    ( 320) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   068   068   010    Pre-fail  Always       -       1510
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       9429
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       6
177 Wear_Leveling_Count     0x0013   099   099   000    Pre-fail  Always       -       11
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   068   068   010    Pre-fail  Always       -       1510
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   068   068   010    Pre-fail  Always       -       1510
187 Reported_Uncorrect      0x0032   099   099   000    Old_age   Always       -       8242
190 Airflow_Temperature_Cel 0x0032   068   056   000    Old_age   Always       -       32
195 Hardware_ECC_Recovered  0x001a   199   199   000    Old_age   Always       -       8242
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0
235 Unknown_Attribute       0x0012   099   099   000    Old_age   Always       -       4
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       35415578583

SMART Error Log Version: 1
ATA Error Count: 8242 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 8242 occurred at disk power-on lifetime: 9429 hours (392 days + 21 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 10 d8 5d e0  Error:

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------

Error 8241 occurred at disk power-on lifetime: 9428 hours (392 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 e8 d7 5d e0  Error:

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------

Error 8240 occurred at disk power-on lifetime: 9427 hours (392 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 e8 d7 5d e0  Error:

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------

Error 8239 occurred at disk power-on lifetime: 9427 hours (392 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 e8 d7 5d e0  Error:

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------

Error 8238 occurred at disk power-on lifetime: 9426 hours (392 days + 18 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 e8 d7 5d e0  Error:

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
  256        0    65535  Read_scanning was never started
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Answer 1

那是一个很差的磁盘。见这里：

[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[Wed Sep 28 12:29:02 2022] sd 0:0:0:0: [sda] CDB: Read(16) 88 00 00 00 00 01 84 5d d8 10 00 00 00 08 00 00
[Wed Sep 28 12:29:02 2022] blk_update_request: I/O error, dev sda, sector 6515709968

这告诉您，操作系统无法从磁盘读取数据。

见这里：

SMART Attributes Data Structure revision number: 1

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   068   068   010    Pre-fail  Always       -       1510
  9 Power_On_Hours          0x0032   098   098   000    Old_age   Always       -       9429
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       6

磁盘已经运行了大约13个月(假设为24x7，只记录了6个电源周期)。不幸的是，1510个重新分配的部门大约比我满意的多1505。

在这里，我们可以看到有很多磁盘读取失败：

183 Runtime_Bad_Block       0x0013   068   068   010    Pre-fail  Always       -       1510
187 Reported_Uncorrect      0x0032   099   099   000    Old_age   Always       -       8242

195 Hardware_ECC_Recovered  0x001a   199   199   000    Old_age   Always       -       8242

您已经编写了相当数量的块(潜在的131 TB)，但没有什么特别之处(SSD在其生命周期中有一个保证的上写期望，而您的期望是2400 TB)，而且肯定不是不成比例的：

241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       35415578583

我有点担心操作系统将磁盘视为有512 B扇区；它们应该是4kB扇区/块。可能值得使用fdisk或parted之类的工具来检查它，看看它是否是smartctl中的一个错误。

最后，我想说是时候换一个新磁盘了。鉴于它在13个月后就死了，我本以为有12个月的保修期，但看来EVOs有五年的保修期。

在评论中，OP问道：“这是否是一种让磁盘继续工作的方法，尽管存在hw问题，比如注销扇区?为什么iostat仍然显示出所有的正常状态？”

如果无法读取磁盘扇区/块，则在OS级别会出现错误。通常情况下，当试图读取(编辑、复制、备份)文件时，它将以IO错误告终。磁盘将将扇区/块标记为不可读，而不可更正的计数将增加。当OS (或应用程序)将数据写入磁盘上的扇区/块时，固件将利用这个机会将扇区/块重新映射到其备用扇区/块中的一个，这些备用扇区/块正是为此目的而预留的。不可修正的计数可能会减少，而重新分配的计数会增加。

到目前为止，磁盘固件已经成功地重新分配了1510个扇区/块，而且大多数IO都很好(这就是为什么iostat看起来还行)。您经常会碰到另一个不可读的扇区/块--如dmesg中所示，除非您尝试给它写信，否则这种情况不会消失。迟早，磁盘将耗尽其保留的扇区/块，您将得到一连串的写入错误，然后是一个损坏的文件系统。这真的不是你想去的地方，所以我建议你试着在保修期内更换磁盘。

无论如何，make确定您已经并将继续进行良好的备份。