CentOS 7没有在英特尔C606 SCU上显示前两个驱动器

Question

我使用的是一个SuperMicro X9SRi-3F主板，它使用英特尔的C606芯片组，具有双口4端口的SAS功能。我使用的HDD是4倍WD Re (WD6001F9YZ)和2倍WD Gold (WD6002FRYZ)。容量均为6TB。

WD6001F9YZ连接到端口0-3，WD6002FRYZ在端口4和5上(应该注意，我最终将向端口6和7添加两个WD6001F9YZ)。

OS安装在一个1TB驱动器上，该驱动器连接到板上的一个普通SATA2.0端口。

CentOS 7可以识别所有驱动器，因为lsblk输出以下内容：

NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda               8:0    0   5.5T  0 disk 
sdb               8:16   0   5.5T  0 disk 
sdc               8:32   0   5.5T  0 disk 
sdd               8:48   0   5.5T  0 disk 
sde               8:64   0   5.5T  0 disk 
sdf               8:80   0   5.5T  0 disk 

它们也都出现在/dev/disk/by-id中：

lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:06 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

但是，只有驱动器c-f出现在/dev/disk/by-path中：

lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe00-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe01-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe02-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-0x500304801349fe03-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 13:42 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdd

在MB的BIOS中，当启用SCU单元和启用"SCU RAID选项ROM/UEFI驱动程序“时，所有驱动器都会被识别：

当启用SCU单元，但禁用ROM选项时，驱动器不会出现在BIOS中：

这两个设置(选项ROM禁用/启用)导致相同的问题。当然，禁用SCU意味着它根本不会出现在操作系统中(从lspci的输出中消失)

lshw -c storage的输出如下所示：

*-sas                     
   description: Serial Attached SCSI controller
   product: C606 chipset Dual 4-Port SATA/SAS Storage Control Unit
   vendor: Intel Corporation
   physical id: 0
   bus info: pci@0000:03:00.0
   logical name: scsi6
   logical name: scsi7
   version: 06
   width: 64 bits
   clock: 33MHz
   capabilities: sas pm pciexpress msix bus_master cap_list
   configuration: driver=isci latency=0
   resources: irq:49 memory:fa8f8000-fa8fffff memory:fa000000-fa7fffff ioport:e100(size=256) ioport:e000(size=256) memory:fa800000-fa8f7fff

这对我来说是个问题，因为我打算设置一个8磁盘的ZFS数组，并希望引用驱动器在系统中的物理位置(slot1、slot2等)。

我的直觉告诉我，这是C606内核驱动程序的问题，但我真的不知道。

编辑

如果我将我打算在端口6和7上使用的其他SATA驱动器热插拔，/dev/disk/by-path如下所示：

lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000001-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000002-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-0x5fcfffff00000003-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-0x5fcfffff00000005-lun-0-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy2-lun-0-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 pci-0000:03:00.0-sas-phy3-lun-0-part2 -> ../../sdj2

以前使用sas-0x500304801349fe##启动的路径部分已经更改为sas-0x5fcfffff000000##，因为我已经禁用了BIOS中的选项ROM。但有趣的是，sdd和sdf已经从by-path目录中消失了，但是sda、sdi和sdj已经出现了。此外，顺序也发生了变化。

当然，所有八个驱动器都显示在by-id目录中：

lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2 -> ../../sdj
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2-part1 -> ../../sdj1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVJK2-part2 -> ../../sdj2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX -> ../../sdi
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX-part1 -> ../../sdi1
lrwxrwxrwx. 1 root root 10 Oct 17 14:24 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVNXX-part2 -> ../../sdi2
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 14:20 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

编辑2

随着每次引导，by-path目录似乎都发生了变化。目前的情况如下：

lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000001-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000002-lun-0 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000003-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-0x5fcfffff00000004-lun-0 -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy0-lun-0 -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy1-lun-0 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy2-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 pci-0000:03:00.0-sas-phy3-lun-0 -> ../../sdd

虽然by-id路径如下所示：

lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA58PYRS -> ../../sda
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV7KR -> ../../sdb
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LV87E -> ../../sdc
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6001F9YZ-09YUWL1_WD-WX41DA5LVL4J -> ../../sdd
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6002FRYZ-01WD5B0_K1HNL2KD -> ../../sde
lrwxrwxrwx. 1 root root  9 Oct 17 15:03 ata-WDC_WD6002FRYZ-01WD5B0_K1JMY3ND -> ../../sdf

磁盘始终被分配相同的设备ID (sda/sdb/etc)，这是令人放心的，但路径的变化是不可预测的(使它完全不能用于我的目的)。

编辑3

du -a /sys/devices/pci0000\:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0 | grep -E 'sd.' | grep -vE 'sd./'的输出显示了驱动器的正确映射：

0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:0/end_device-6:0/target6:0:0/6:0:0:0/block/sda
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:1/end_device-6:1/target6:0:1/6:0:1:0/block/sdb
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:2/end_device-6:2/target6:0:2/6:0:2:0/block/sdc
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host6/port-6:3/end_device-6:3/target6:0:3/6:0:3:0/block/sdd
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host7/port-7:0/end_device-7:0/target7:0:0/7:0:0:0/block/sde
0   /sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0/host7/port-7:1/end_device-7:1/target7:0:1/7:0:1:0/block/sdf

所以我认为这个问题与udev中的一个bug有关

Answer 1

我还有一个戴尔T420服务器，udevadm info的输出给了我以下内容：

E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:0/0:0:0:0/block/sda
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:1/0:0:1:0/block/sdb
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:2/0:0:2:0/block/sdc
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:3/0:0:3:0/block/sdd
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:4/0:0:4:0/block/sde
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:5/0:0:5:0/block/sdf
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:6/0:0:6:0/block/sdg
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:08:00.0/host0/target0:0:7/0:0:7:0/block/sdh

如您所见，设备路径(host0)中只有host#的实例，而英特尔SCU有两个实例(host6和host7)。显然，CentOS 7上的udev不知道如何正确处理这个问题，只是简单地覆盖了设备链接(因此，host6节点下的任何设备都会在/dev/disks/by-path中获得其符号链接，由host7节点下的相应设备覆盖)。

看来我现在需要学习怎么写udev规则了..。

编辑

首次尝试使用udev规则来解决问题：https://gist.github.com/dghodgson/49da6175371cdde317e662fb8a7d078a

它很丑，而且很笨重。根本无法正确处理热插拔，也不会对分区做任何事情。我需要找到一种从现有信息创建更新属性的方法，而不是就地编辑它们，否则，每次重新加载udev规则时，路径都有可能发生更改。

编辑2

Gist已经更新为输出路径，类似于handle_scsi_default函数在path_id内置中产生的输出路径。它现在要可靠得多，而且还可以处理分区。希望其他人也会发现它是有用的。不过这还是个黑客所以YMMV。

目前正致力于在udev中进行适当的修复。