软件RAID5中的硬盘销毁

Question

我的盒子里有6个HDD，2x256Gb ATA OCZ-AGILITY4 4 SSD。

• 每个块被划分为一个48 of的块和一个208 of的块。
• 48 as分区是RAID0条带化的，用作交换空间(md0)。
• 208 as分区是RAID1条带化的，用作/文件系统(md1)。

还有4x3.0TbATA ST3000DM001-9YN166驱动器，每个驱动器都有一个分区，这4个分区是RAID5条带(md128)。所有这些都是ext4格式的，运行ubuntu12.04服务器。sda和sdb是SSD，而sdc、sdd、sde和sdf是HDD。

据我所知(据我所知)，md128变成了只读。对于事件，sys日志如下所示：

Aug 23 16:25:24 crick kernel: [617040.416257] ata4.00: exception Emask 0x0 SAct 0x1f SErr 0x0 action 0x0
Aug 23 16:25:24 crick kernel: [617040.416260] ata4.00: irq_stat 0x40000008
Aug 23 16:25:24 crick kernel: [617040.416262] ata4.00: failed command: READ FPDMA QUEUED
Aug 23 16:25:24 crick kernel: [617040.416265] ata4.00: cmd 60/08:08:00:af:cc/00:00:9c:00:00/40 tag 1 ncq 4096 in
Aug 23 16:25:24 crick kernel: [617040.416265]          res 41/40:08:00:af:cc/00:00:9c:00:00/00 Emask 0x409 (media error) <F>
Aug 23 16:25:24 crick kernel: [617040.416266] ata4.00: status: { DRDY ERR }
Aug 23 16:25:24 crick kernel: [617040.416267] ata4.00: error: { UNC }
Aug 23 16:25:24 crick kernel: [617040.417510] ata4.00: configured for UDMA/133
Aug 23 16:25:24 crick kernel: [617040.417527] sd 3:0:0:0: [sdd] Unhandled sense code
Aug 23 16:25:24 crick kernel: [617040.417528] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417529] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 23 16:25:24 crick kernel: [617040.417530] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417531] Sense Key : Medium Error [current] [descriptor]
Aug 23 16:25:24 crick kernel: [617040.417533] Descriptor sense data with sense descriptors (in hex):
Aug 23 16:25:24 crick kernel: [617040.417534]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
Aug 23 16:25:24 crick kernel: [617040.417538]         9c cc af 00 
Aug 23 16:25:24 crick kernel: [617040.417540] sd 3:0:0:0: [sdd]  
Aug 23 16:25:24 crick kernel: [617040.417541] Add. Sense: Unrecovered read error - auto reallocate failed
Aug 23 16:25:24 crick kernel: [617040.417542] sd 3:0:0:0: [sdd] CDB: 
Aug 23 16:25:24 crick kernel: [617040.417543] Read(10): 28 00 9c cc af 00 00 00 08 00
Aug 23 16:25:24 crick kernel: [617040.417547] end_request: I/O error, dev sdd, sector 2630659840
Aug 23 16:25:24 crick kernel: [617040.417550] md/raid:md128: read error not correctable (sector 2630657792 on sdd1).
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Disk failure on sdd1, disabling device.
Aug 23 16:25:24 crick kernel: [617040.417552] md/raid:md128: Operation continuing on 2 devices.
Aug 23 16:25:24 crick kernel: [617040.417563] ata4: EH complete
Aug 23 16:25:25 crick kernel: [617040.455605] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.455609]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.455610]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.455611]  disk 1, o:0, dev:sdd1
Aug 23 16:25:25 crick kernel: [617040.455612]  disk 2, o:1, dev:sde1
Aug 23 16:25:25 crick kernel: [617040.489941] RAID conf printout:
Aug 23 16:25:25 crick kernel: [617040.489945]  --- level:5 rd:4 wd:2
Aug 23 16:25:25 crick kernel: [617040.489947]  disk 0, o:1, dev:sdc1
Aug 23 16:25:25 crick kernel: [617040.489948]  disk 2, o:1, dev:sde1

A slew of:
Aug 23 16:25:25 crick kernel: [617040.539926] Buffer I/O error on device md128, logical block 986401023
with different block addresses then

Aug 23 16:25:25 crick kernel: [617040.539929] EXT4-fs warning (device md128): ext4_end_bio:248: I/O error writing to inode 42993727 (offset 11551637504 size 524288 starting block 986400896)
Aug 23 16:25:25 crick kernel: [617040.541690] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.541707] Aborting journal on device md128-8.
Aug 23 16:25:25 crick kernel: [617040.541720] EXT4-fs error (device md128) in ext4_free_blocks:4702: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.541727] Buffer I/O error on device md128, logical block 1098416128
Aug 23 16:25:25 crick kernel: [617040.541729] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541734] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.541736] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.541740] JBD2: Error -5 detected when updating journal superblock for md128-8.
Aug 23 16:25:25 crick kernel: [617040.541743] EXT4-fs (md128): delayed block allocation failed for inode 49152114 at logical offset 994912 with max blocks 2048 with error -30
Aug 23 16:25:25 crick kernel: [617040.541745] EXT4-fs (md128): This should not happen!! Data will be lost
Aug 23 16:25:25 crick kernel: [617040.541745] 
Aug 23 16:25:25 crick kernel: [617040.541806] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542518] EXT4-fs error (device md128) in ext4_da_writepages:2390: Journal has aborted
Aug 23 16:25:25 crick kernel: [617040.542526] JBD2: Detected IO errors while flushing file data on md128-8
Aug 23 16:25:25 crick kernel: [617040.542529] Buffer I/O error on device md128, logical block 0
Aug 23 16:25:25 crick kernel: [617040.542531] lost page write due to I/O error on md128
Aug 23 16:25:25 crick kernel: [617040.542798] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal
Aug 23 16:25:25 crick kernel: [617040.542810] EXT4-fs (md128): Remounting filesystem read-only
Aug 23 16:25:25 crick kernel: [617040.542815] EXT4-fs (md128): previous I/O error to superblock detected
Aug 23 16:25:25 crick kernel: [617040.542835] EXT4-fs (md128): I/O error while writing superblock
Aug 23 16:25:25 crick kernel: [617040.542838] EXT4-fs (md128): ext4_da_writepages: jbd2_start: 15984 pages, ino 49152114; err -30
Aug 23 16:25:25 crick kernel: [617040.544887] EXT4-fs error (device md128): ext4_journal_start_sb:371: Detected aborted journal

当我使用磁盘实用程序或gparted时，它甚至不再被检测到。因此，md128被降级。我所做的第一件事是通过磁盘实用程序卸载它，然后我尝试查看是否还有其他东西可以找到它：

fdisk -l

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a88fe

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1       406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sda2            2046   406368255   203183105    5  Extended
/dev/sda5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

Disk /dev/md0: 96.0 GB, 95998181376 bytes
2 heads, 4 sectors/track, 23437056 cylinders, total 187496448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 207.9 GB, 207925149696 bytes
2 heads, 4 sectors/track, 50762976 cylinders, total 406103808 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/sdb: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000b0740

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *   406368256   500117503    46874624   fd  Linux raid autodetect
/dev/sdb2            2046   406368255   203183105    5  Extended
/dev/sdb5            2048   406368255   203183104   fd  Linux raid autodetect

Partition table entries are not in disk order

WARNING: GPT (GUID Partition Table) detected on '/dev/sdc'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

Disk /dev/md128: 9001.4 GB, 9001370124288 bytes
2 heads, 4 sectors/track, -2097367168 cylinders, total 17580801024 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1572864 bytes
Disk identifier: 0x00000000

Disk /dev/md128 doesn't contain a valid partition table

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

cat /etc/fstab

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md127 during installation
UUID=b63b7341-0b85-40ed-a67b-acfe4f65f563 /               ext4    errors=remount-ro 0       1
# /genome was on /dev/md128 during installation
UUID=bd6b54be-12ca-479d-879e-8d788fa9d039 /genome         ext4    defaults        0       2
# swap was on /dev/md126 during installation
UUID=0470550a-6e92-485d-ad41-665c3f313287 none            swap    sw              0       0

~# cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md128 : active raid5 sde1[2] sdc1[0]
      8790400512 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/2] [U_U_]

md1 : active raid1 sdb5[1] sda5[0]
      203051904 blocks super 1.2 [2/2] [UU]

md0 : active raid0 sdb1[1] sda1[0]
      93748224 blocks super 1.2 512k chunks

unused devices: <none>

~# mdadm --细节扫描

ARRAY /dev/md/0 metadata=1.2 name=crick:0 UUID=d0d08eab:a7e54021:25973acb:10dd5fba
ARRAY /dev/md/1 metadata=1.2 name=crick:1 UUID=e2774945:5b3ee3eb:2ad9390f:35153b82
ARRAY /dev/md/128 metadata=1.2 name=crick:128 UUID=345cb755:0ae1c919:d98a45ca:1baf3364

不过，我对linux的理解并不是最好的。据我所知，数组超级块可以看到它所构建的sdf的UUID，但是(无论出于什么原因) sdf没有被检测到。这意味着RAID数组被破坏了，因此，作为安全防范措施，ubuntu将文件系统设置为RO。我仍然可以在/没有问题的情况下打开、编辑和创建文件，操作系统仍然运行完美，我只是无法访问md128上的数据。我还没有关掉这台机器来拆下驱动器看看它是否坏了。

大约一周前，也发生了同样的问题(随机切换到RO)，只是在那个时候它击中了md1 (SSD)。当这种情况发生时，超级块在md128上被擦除。我很幸运地重新安装了操作系统，并让它使用现有的分区(IE没有格式化分区)重新构建RAID，并设法恢复99.9%的数据(发生时被访问的文件已损坏)。

• 到目前为止，md128的问题是我设法解决的一个问题，还是一个我应该担心的更深层次问题的迹象？
• 从这里继续前进的最佳方式是什么:替换sdf并将其重新划分为md128，替换sdf并删除/重建md128或其他什么东西？

编辑:对不起，我是论坛新手，没有看格式说明，我现在已经更正了

Answer 1

很可能是自卫队不久前去世了，你没注意到。现在sdd有一些不好的部门。读取失败导致文件系统切换到只读。您需要替换失败的驱动器，并对文件系统进行fsck。