fio配置以根据提供者SLA度量IOPS

Question

因此，一个提供商给了我们500个IOPS/TB作为他们在VMWare & RAID5-SAN环境中磁盘性能的SLA标准。这显然是由以下几个方面衡量的：

• 16平均传输块大小
• 3:1读:写比
• 多线程IO操作
• 80%随机IO建模
• 读取缓存命中率为20%

我想要做的是确定是否有任何特定的Linux获得这种性能，然后与其他提供者运行相同的基准，以便进行比较。

环顾四周，菲奥似乎是衡量上述情况的最可配置的。到目前为止，我得到的配置是：

[global]
blocksize=16k
rwmixread=75     # 3:1 read:write ratio
ramp_time=30
runtime=600
time_based
buffered=1
# size = free-ram * 80% / 5
# so we get a ~20% cache hit across the 5x processes
# this is for an 8GB ram host with 7.3GB free after buffers/cache
size=1180m

# create a mix to get to 80% random reads
# also means we'll be doing at least 5x IO operations in parallel
[sla-0]
readwrite=randrw:2

[sla-1]
readwrite=randrw:2

[sla-2]
readwrite=randrw

[sla-3]
readwrite=randrw

[sla-4]
readwrite=randrw

关于改进的建议？使用buffered和默认的ioengine是最好的方法吗？

如果我在一个拥有8GB内存和470 8GB内存的空闲4x虚拟机上运行这个程序，我希望通过上面的(500 * 0.47)获得235个IOPS。我得到的结果是：

sla-0: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2
sla-1: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2
sla-2: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2
sla-3: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2
sla-4: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2
Starting 5 processes
sla-0: Laying out IO file(s) (1 file(s) / 1180MB)
sla-1: Laying out IO file(s) (1 file(s) / 1180MB)
sla-2: Laying out IO file(s) (1 file(s) / 1180MB)
sla-3: Laying out IO file(s) (1 file(s) / 1180MB)
sla-4: Laying out IO file(s) (1 file(s) / 1180MB)
Jobs: 5 (f=5): [mmmmm] [100.0% done] [5931K/1966K /s] [362/120 iops] [eta 00m:00s] 
sla-0: (groupid=0, jobs=1): err= 0: pid=16701
  read : io=1086MB, bw=1853KB/s, iops=115, runt=600003msec
    clat (usec): min=4, max=1771K, avg=8607.53, stdev=22114.44
    bw (KB/s) : min=    0, max= 4087, per=24.44%, avg=1914.96, stdev=1130.29
  write: io=372416KB, bw=635586B/s, iops=38, runt=600003msec
    clat (usec): min=6, max=2574, avg=57.38, stdev=79.65
    bw (KB/s) : min=    0, max=11119, per=26.07%, avg=679.63, stdev=517.84
  cpu          : usr=0.08%, sys=0.63%, ctx=64513, majf=0, minf=109
  IO depths    : 1=107.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=69474/23276, short=0/0
     lat (usec): 10=10.23%, 20=8.89%, 50=4.15%, 100=11.66%, 250=0.83%
     lat (usec): 500=1.48%, 750=1.41%, 1000=0.82%
     lat (msec): 2=0.83%, 4=1.56%, 10=47.07%, 20=5.91%, 50=4.24%
     lat (msec): 100=0.55%, 250=0.29%, 500=0.06%, 750=0.01%, 1000=0.01%
     lat (msec): 2000=0.01%
sla-1: (groupid=0, jobs=1): err= 0: pid=16702
  read : io=963360KB, bw=1605KB/s, iops=100, runt=600180msec
    clat (usec): min=4, max=2396K, avg=9934.23, stdev=30986.37
    bw (KB/s) : min=    0, max= 4657, per=21.64%, avg=1695.89, stdev=1273.00
  write: io=326000KB, bw=556206B/s, iops=33, runt=600180msec
    clat (usec): min=6, max=3882, avg=55.07, stdev=77.92
    bw (KB/s) : min=    0, max=10708, per=23.74%, avg=618.92, stdev=559.01
  cpu          : usr=0.08%, sys=0.53%, ctx=55500, majf=0, minf=129
  IO depths    : 1=108.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=60210/20375, short=0/0
     lat (usec): 10=11.36%, 20=9.63%, 50=3.56%, 100=11.97%, 250=0.81%
     lat (usec): 500=0.66%, 750=0.50%, 1000=0.37%
     lat (msec): 2=0.33%, 4=0.74%, 10=49.56%, 20=3.78%, 50=5.48%
     lat (msec): 100=0.60%, 250=0.43%, 500=0.16%, 750=0.04%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%
sla-2: (groupid=0, jobs=1): err= 0: pid=16703
  read : io=827584KB, bw=1379KB/s, iops=86, runt=600012msec
    clat (usec): min=397, max=2396K, avg=11569.59, stdev=31237.03
    bw (KB/s) : min=    0, max= 4237, per=18.60%, avg=1457.59, stdev=1113.89
  write: io=276192KB, bw=471358B/s, iops=28, runt=600012msec
    clat (usec): min=8, max=8339, avg=63.95, stdev=121.52
    bw (KB/s) : min=    0, max= 8531, per=20.52%, avg=534.85, stdev=478.91
  cpu          : usr=0.07%, sys=0.54%, ctx=57019, majf=0, minf=89
  IO depths    : 1=109.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=51724/17262, short=0/0
     lat (usec): 10=0.98%, 20=5.38%, 50=3.53%, 100=13.68%, 250=0.92%
     lat (usec): 500=0.60%, 750=0.39%, 1000=0.22%
     lat (msec): 2=0.24%, 4=2.26%, 10=59.15%, 20=4.90%, 50=6.28%
     lat (msec): 100=0.78%, 250=0.48%, 500=0.18%, 750=0.03%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%
sla-3: (groupid=0, jobs=1): err= 0: pid=16704
  read : io=865920KB, bw=1443KB/s, iops=90, runt=600005msec
    clat (usec): min=369, max=2396K, avg=11052.97, stdev=32396.85
    bw (KB/s) : min=    0, max= 5984, per=19.47%, avg=1525.97, stdev=1164.42
  write: io=285568KB, bw=487365B/s, iops=29, runt=600005msec
    clat (usec): min=7, max=11910, avg=65.72, stdev=154.09
    bw (KB/s) : min=    0, max=11064, per=21.38%, avg=557.30, stdev=534.59
  cpu          : usr=0.07%, sys=0.57%, ctx=59458, majf=0, minf=109
  IO depths    : 1=109.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=54120/17848, short=0/0
     lat (usec): 10=0.99%, 20=5.11%, 50=3.58%, 100=13.64%, 250=0.89%
     lat (usec): 500=0.71%, 750=0.48%, 1000=0.30%
     lat (msec): 2=0.70%, 4=4.00%, 10=57.63%, 20=5.21%, 50=5.40%
     lat (msec): 100=0.70%, 250=0.43%, 500=0.16%, 750=0.03%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%
sla-4: (groupid=0, jobs=1): err= 0: pid=16705
  read : io=934752KB, bw=1558KB/s, iops=97, runt=600007msec
    clat (usec): min=187, max=2396K, avg=10236.87, stdev=26080.98
    bw (KB/s) : min=    0, max=11419, per=20.74%, avg=1625.28, stdev=1338.26
  write: io=304528KB, bw=519721B/s, iops=31, runt=600007msec
    clat (usec): min=7, max=7572, avg=67.29, stdev=117.27
    bw (KB/s) : min=    0, max=10772, per=22.06%, avg=575.17, stdev=560.68
  cpu          : usr=0.08%, sys=0.60%, ctx=63685, majf=0, minf=129
  IO depths    : 1=108.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued r/w: total=58422/19033, short=0/0
     lat (usec): 10=0.81%, 20=4.77%, 50=3.62%, 100=13.77%, 250=0.97%
     lat (usec): 500=1.45%, 750=0.64%, 1000=0.53%
     lat (msec): 2=1.75%, 4=4.71%, 10=53.48%, 20=6.92%, 50=5.53%
     lat (msec): 100=0.56%, 250=0.37%, 500=0.08%, 750=0.02%, 1000=0.01%
     lat (msec): 2000=0.01%, >=2000=0.01%

Run status group 0 (all jobs):
   READ: io=4593MB, aggrb=7836KB/s, minb=1412KB/s, maxb=1897KB/s, mint=600003msec, maxt=600180msec
  WRITE: io=1528MB, aggrb=2607KB/s, minb=471KB/s, maxb=635KB/s, mint=600003msec, maxt=600180msec

Disk stats (read/write):
  dm-0: ios=298995/596154, merge=0/0, ticks=3107720/433061790, in_queue=436170340, util=99.68%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%

总结每一项工作的读、写IOPS (为什么fio不将其包括在摘要中？)我得到了647，这似乎超过了他们指定的服务水平。任何明显的东西我都遗漏了，或者它们的指标对某些工作负载有很大的偏差(特别是我对数据仓库工作负载的PostgreSQL感兴趣)。

Answer 1

SQL和数据仓库更像是8:1读到写，都是小块，都是随机的。在任何情况下，除了随机读取之外，任何东西都很容易缓存，而且不太可能导致磁盘性能问题。在不知道磁盘是如何工作的情况下，很难真正帮助他们，但是考虑一下，当他们指定"RAID5-SAN环境“时，问他们是什么意思。

由于它们将SLA指定为每个TB的IOPS，因此我猜测它们提供给您的每个卷都应该位于单独的RAID-5中，允许在添加卷时使用更多的IOPS。不好的性能很容易由不好的raid邻居造成:与您相同的raid上的卷占用的存储资源超过了它们的公平份额。这方面的问题是，有时您的SLA将被超过，但有时您将不得不处理高延迟。

首先要警告他们你对性能不满意，他们可能会把你转移到一个更低利用率的raid上，这样就可以解决你所有的问题。同时询问他们是否有raid-10的存储空间，并且可能要求提供一个卷，而不是raid 5。如果问题再次出现，那么考虑获得自己的存储空间，或者找到其他可以为您提供更好性能的主机。