使用mdadm软件ext4 RAID1 8 Dell PowerEdge T330服务器在未加密和LUKS加密的PowerEdge文件系统上的慢速写入速度(<2MB/S)

Question

我们已经在一个新的戴尔PowerEdge T330上安装了Debian 8，在使用mdadm的RAID1数组中有两个分区/和/var。在测试期间，主要应用程序: mysql和tomcat被停止。我们正在从两个分区获得糟糕的写入性能，尽管读取性能足够。这是两个相同的服务器中的一个以相同的方式设置的观察结果。任何帮助都将不胜感激。

性能

root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data & dd bs=16k \ count=102400 oflag=direct if=/dev/ of=$FILE && rm $FILE & FILE=/var/tmp/test_data && dd bs=16k \ count=102400 oflag=direct if=/dev/零of=$FILE & rm $FILE

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 886.418 s, 1.9 MB/s

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 894.832 s, 1.9 MB/s

root@bcmdit-519:/home/bcmdit# hdparm -t /dev/sda；hdparm -t /dev/sdb；hdparm -t /dev/md0；hdparm -t /dev/md1 1

/dev/sda:

    Timing buffered disk reads: 394 MB in  3.00 seconds = 131.15 MB/sec

/dev/sdb:

    Timing buffered disk reads: 394 MB in  3.01 seconds = 131.05 MB/sec

/dev/md0:

    Timing buffered disk reads: 398 MB in  3.00 seconds = 132.45 MB/sec

/dev/md1:

    Timing buffered disk reads: 318 MB in  3.00 seconds = 106.00 MB/sec

<#>引用

https://severfault.com/questions/832117/how-increase-write-speed-of-raid1-mdadm https://wiki.mikejung.biz/Software_RAID 在RAID1 1上写访问时间慢 https://bbs.archlinux.org/viewtopic.php?id=173791等。

配置

加密是使用以下方法设置的：

root@bcmdit-519:/home/bcmdit# cryptsetup luksDump -E

Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1

配置片段

root@bcmdit-519:/home/bcmdit# facter virtual productname lsbdistid \
                     lsbdistrelease processor0 blockdevice_sda_model \  
                     blockdevice_sdb_model bios_version && uname -a && uptime
----------

    bios_version => 2.4.3
    blockdevice_sda_model => ST1000NX0423
    blockdevice_sdb_model => ST1000NX0423
    lsbdistid => Debian
    lsbdistrelease => 8.10
    processor0 => Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
    productname => PowerEdge T330
    virtual => physical
    Linux bcmdit-519 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux
     14:45:58 up  2:49,  2 users,  load average: 0.06, 0.17, 0.44

 root@bcmdit-519:/home/bcmdit# grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub

    GRUB_CMDLINE_LINUX_DEFAULT="quiet erst_disable elevator=deadline"

root@bcmdit-519:/home/bcmdit# free -m         

             total       used       free     shared    buffers     cached
Mem:         32202       1532      30670          9         17        369
-/+ buffers/cache:       1145      31056
Swap:            0          0          0

root@bcmdit-519:/home/bcmdit# parted /dev/sda打印

    Model: ATA ST1000NX0423 (scsi)
    Disk /dev/sda: 1000GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size   Type      File system  Flags
     1      1049kB  500GB   500GB  primary                boot, raid
     2      500GB   1000GB  500GB  extended
     5      500GB   1000GB  500GB  logical                raid

root@bcmdit-519:/home/bcmdit# parted /dev/sdb打印

    Model: ATA ST1000NX0423 (scsi)
    Disk /dev/sdb: 1000GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size   Type      File system  Flags
     1      1049kB  500GB   500GB  primary                raid
     2      500GB   1000GB  500GB  extended
     5      500GB   1000GB  500GB  logical                raid

----------

root@bcmdit-519:/home/bcmdit# cat /proc/mdstat

    Personalities : [raid1] 
    md1 : active raid1 sda5[0] sdb5[1]
          488249344 blocks super 1.2 [2/2] [UU]
          bitmap: 3/4 pages [12KB], 65536KB chunk

    md0 : active raid1 sda1[0] sdb1[1]
          488248320 blocks super 1.2 [2/2] [UU]
          bitmap: 2/4 pages [8KB], 65536KB chunk

    unused devices: 

root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md0

    /dev/md0:
            Version : 1.2
      Creation Time : Mon Apr 16 13:46:51 2018
         Raid Level : raid1
         Array Size : 488248320 (465.63 GiB 499.97 GB)
      Used Dev Size : 488248320 (465.63 GiB 499.97 GB)
       Raid Devices : 2
      Total Devices : 2
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Tue May 15 14:26:47 2018
              State : clean 
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0

               Name : bcmdit-519:0  (local to host bcmdit-519)
               UUID : afd3968c:2e8b191d:4504f21e:255b6470
             Events : 1703

        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1

 root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md1

    /dev/md1: 

            Version : 1.2
      Creation Time : Mon Apr 16 13:47:06 2018
         Raid Level : raid1
         Array Size : 488249344 (465.63 GiB 499.97 GB)
      Used Dev Size : 488249344 (465.63 GiB 499.97 GB)
       Raid Devices : 2
      Total Devices : 2
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Tue May 15 14:15:11 2018
              State : active 
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0

               Name : bcmdit-519:1  (local to host bcmdit-519)
               UUID : e46f968a:e8fff775:ecee9cfb:4ad88574
             Events : 2659

        Number   Major   Minor   RaidDevice State
           0       8        5        0      active sync   /dev/sda5
           1       8       21        1      active sync   /dev/sdb5

root@bcmdit-519:/home/bcmdit# cat /etc/crypttab

    crypt1 UUID=1e7b64ac-f187-4fac-9712-8e0dacadfca7 /root/.crypt1 luks

root@bcmdit-519:/home/bcmdit# grep -v '^#‘/etc/

    UUID=c6baa173-8ea6-4598-a965-eee728a93d69 /               ext4    defaults,errors=remount-ro 0       1
    /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
    /dev/mapper/crypt1 /var ext4 defaults,errors=remount-ro 0       2
    /var/swapfile1 none swap sw,nofail 0       0

root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sda\ -n 20

    smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Device Model:     ST1000NX0423
    Serial Number:    W4713QXE
    LU WWN Device Id: 5 000c50 0abb06247
    Add. Product Id:  DELL(tm)
    Firmware Version: NA07
    User Capacity:    1,000,204,886,016 bytes [1.00 TB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    7200 rpm
    Form Factor:      2.5 inches
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   ACS-3 (minor revision not indicated)
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Tue May 15 14:29:03 2018 PDT
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sdb\ -n 20

    smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Device Model:     ST1000NX0423
    Serial Number:    W4714VDQ
    LU WWN Device Id: 5 000c50 0abf99927
    Add. Product Id:  DELL(tm)
    Firmware Version: NA07
    User Capacity:    1,000,204,886,016 bytes [1.00 TB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    7200 rpm
    Form Factor:      2.5 inches
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   ACS-3 (minor revision not indicated)
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Tue May 15 14:29:11 2018 PDT
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

Update 1

有16M

root@bcmdit-519:/tmp# FILE=/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE \
&& FILE=/var/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 16.6394 s, 103 MB/s
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 17.8649 s, 95.8 MB/s

Update 2找到SMART的希捷驱动器序列号表示企业级驱动器：https://www.cnet.com/products/seagate-enterprise-capacity-2-5-hdd-v-3-1tb-sata-512n/specs/

Update 3我发现驱动器写缓存已关闭，但通过将其设置为on：

hdparm -W1 /dev/sd*

我现在在bs=16k上得到了更好的结果

root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data && dd bs=16k count=102400 oflag=direct if=/dev/zero of=$FILE && rm $FILE
102400+0 records in         
102400+0 records out
1677721600 bytes (1.7 GB) copied, 14.0708 s, 119 MB/s

Update 4

root@ecm-oscar-519:/home/bcmdit# cryptsetup基准测试

# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1394382 iterations per second
PBKDF2-sha256     923042 iterations per second
PBKDF2-sha512     728177 iterations per second
PBKDF2-ripemd160  804122 iterations per second
PBKDF2-whirlpool  313569 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b  1149.9 MiB/s  3655.8 MiB/s
 serpent-cbc   128b    99.6 MiB/s   743.4 MiB/s
 twofish-cbc   128b   219.0 MiB/s   400.0 MiB/s
     aes-cbc   256b   867.5 MiB/s  2904.5 MiB/s
 serpent-cbc   256b    99.6 MiB/s   742.6 MiB/s
 twofish-cbc   256b   218.9 MiB/s   399.8 MiB/s
     aes-xts   256b  3615.1 MiB/s  3617.3 MiB/s
 serpent-xts   256b   710.8 MiB/s   705.0 MiB/s
 twofish-xts   256b   388.1 MiB/s   394.5 MiB/s
     aes-xts   512b  2884.9 MiB/s  2888.1 MiB/s
 serpent-xts   512b   710.7 MiB/s   704.7 MiB/s
 twofish-xts   512b   388.0 MiB/s   394.3 MiB/s

Answer 1

当您向dd请求bs=16K和oflag=direct时，您要求的是许多小型的写操作，这是HDD不好的地方，也是SSD擅长的地方。

您可以使用LVMCache来获得两者的好处(可达到SSD大小)

如果您使用bs=16M或不使用oflag，则写入将被拆分/组合/缓存在内存中，并以最佳大小写入。

为什么dd对磁盘的直接写入速度比对文件的写入慢.

例如；

> dd if=/dev/zero of=test.bin bs=16k count=1000 oflag=direct
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 3.19453 s, 5.1 MB/s

> dd if=/dev/zero of=test.bin bs=16M count=1 oflag=direct
1+0 records in
1+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.291366 s, 57.6 MB/s

> dd if=/dev/zero of=test.bin bs=16k count=1000
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 0.0815558 s, 201 MB/s

> uname -r
4.14.41-130

Answer 2

使用pv(手册页)实用程序如下所示：

pv --average-rate < /dev/urandom > /mdX-MountPoint/SomeFileName

可能被证明是比dd(手册页)更有效的测量速度的方法。

我将输入文件编辑为随机数据，这比静态零更好。

dd(手册页)的问题是必须调整块大小。

对于pv(手册页)，情况并非如此，因为它为最佳吞吐量建立了最大速度。