Iscsi/tests

fio

for more info on fio see: TBD

apt-get install fio

create /root/fio.cfg :

echo "# from mir pve forum
[global]
description=Emulation of Intel IOmeter File Server Access Pattern

[iometer]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=80
direct=1
size=4g
ioengine=libaio
# IOMeter defines the server loads as the following:
# iodepth=1    Linear
# iodepth=4    Very Light
# iodepth=8    Light
# iodepth=64    Moderate
# iodepth=256    Heavy
iodepth=64" > /root/fio.cfg

sys4 tests

napp-it/omnios , motherboard: supermicro x9xcl-f , memory 28GB , cpu:

two SAS2008 IT mode HBA cards

the following tests use a debian jessie virtual machines.

raidz1 with a log drive

zfs: raidz1 : 5 intel ssd pro series 2500 480GB + zil intel ssd s3700

TBD copy and paste from forum thread.

lxc

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [252.7MB/65052KB/0KB /s] [59.3K/14.8K/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=2513: Sat Apr  9 15:24:14 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=303130KB/s, iops=49742, runt= 11060msec
  slat (usec): min=2, max=588, avg= 7.45, stdev=18.05
  clat (usec): min=160, max=252974, avg=885.66, stdev=2765.50
  lat (usec): min=167, max=252977, avg=893.38, stdev=2765.43
  clat percentiles (usec):
  |  1.00th=[  334],  5.00th=[  406], 10.00th=[  482], 20.00th=[  580],
  | 30.00th=[  652], 40.00th=[  716], 50.00th=[  772], 60.00th=[  836],
  | 70.00th=[  892], 80.00th=[  980], 90.00th=[ 1128], 95.00th=[ 1304],
  | 99.00th=[ 1832], 99.50th=[ 2320], 99.90th=[20608], 99.95th=[38656],
  | 99.99th=[136192]
  bw (KB  /s): min=118739, max=444443, per=100.00%, avg=305665.86, stdev=75220.28
  write: io=841688KB, bw=76102KB/s, iops=12445, runt= 11060msec
  slat (usec): min=3, max=1020, avg= 9.17, stdev=20.35
  clat (usec): min=395, max=264907, avg=1552.83, stdev=5218.00
  lat (usec): min=402, max=264915, avg=1562.29, stdev=5217.88
  clat percentiles (usec):
  |  1.00th=[  644],  5.00th=[  804], 10.00th=[  908], 20.00th=[ 1032],
  | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384],
  | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1864], 95.00th=[ 2128],
  | 99.00th=[ 2960], 99.50th=[ 4320], 99.90th=[61696], 99.95th=[121344],
  | 99.99th=[252928]
  bw (KB  /s): min=28575, max=111024, per=100.00%, avg=76804.24, stdev=18922.29
  lat (usec) : 250=0.01%, 500=9.42%, 750=27.66%, 1000=31.51%
  lat (msec) : 2=29.46%, 4=1.58%, 10=0.12%, 20=0.06%, 50=0.12%
  lat (msec) : 100=0.02%, 250=0.03%, 500=0.01%
  cpu  : usr=12.19%, sys=56.86%, ctx=17722, majf=0, minf=8
  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
  submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
  issued  : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
  latency  : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: io=3274.5MB, aggrb=303130KB/s, minb=303130KB/s, maxb=303130KB/s, mint=11060msec, maxt=11060msec
  WRITE: io=841688KB, aggrb=76101KB/s, minb=76101KB/s, maxb=76101KB/s, mint=11060msec, maxt=11060msec

Disk stats (read/write):
  dm-2: ios=543761/136160, merge=0/0, ticks=349496/178408, in_queue=527996, util=99.14%, aggrios=547260/137478, aggrmerge=2955/219, aggrticks=351272/179892, aggrin_queue=530992, aggrutil=98.92%
  sdk: ios=547260/137478, merge=2955/219, ticks=351272/179892, in_queue=530992, util=98.92%

4 drive mirror, without a log drive

4 intel ssd model ___ , no zil

lxc

3032-jessie  ~ # fio /root/fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [93189KB/23350KB/0KB /s] [21.7K/5469/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1857: Tue Apr 12 09:39:34 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=109922KB/s, iops=18037, runt= 30500msec
    slat (usec): min=2, max=8635, avg= 6.00, stdev=16.12
    clat (usec): min=144, max=43100, avg=2123.05, stdev=2388.69
     lat (usec): min=166, max=43103, avg=2129.27, stdev=2388.60
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  418], 10.00th=[  486], 20.00th=[  604],
     | 30.00th=[  732], 40.00th=[  892], 50.00th=[ 1160], 60.00th=[ 1624],
     | 70.00th=[ 2384], 80.00th=[ 3312], 90.00th=[ 4896], 95.00th=[ 6496],
     | 99.00th=[10816], 99.50th=[13248], 99.90th=[22144], 99.95th=[24192],
     | 99.99th=[31616]
    bw (KB  /s): min=79049, max=218569, per=100.00%, avg=110088.87, stdev=30276.89
  write: io=841688KB, bw=27596KB/s, iops=4512, runt= 30500msec
    slat (usec): min=3, max=8921, avg= 7.78, stdev=35.57
    clat (usec): min=462, max=54371, avg=5654.28, stdev=3658.82
     lat (usec): min=469, max=54379, avg=5662.30, stdev=3658.89
    clat percentiles (usec):
     |  1.00th=[ 1032],  5.00th=[ 1448], 10.00th=[ 1880], 20.00th=[ 2704],
     | 30.00th=[ 3408], 40.00th=[ 4128], 50.00th=[ 4896], 60.00th=[ 5728],
     | 70.00th=[ 6752], 80.00th=[ 8096], 90.00th=[10304], 95.00th=[12352],
     | 99.00th=[17536], 99.50th=[21632], 99.90th=[28288], 99.95th=[31104],
     | 99.99th=[41216]
    bw (KB  /s): min=19175, max=53703, per=100.00%, avg=27661.45, stdev=7629.12
    lat (usec) : 250=0.09%, 500=8.83%, 750=16.25%, 1000=10.74%
    lat (msec) : 2=18.74%, 4=21.13%, 10=20.97%, 20=2.96%, 50=0.31%
    lat (msec) : 100=0.01%
  cpu          : usr=5.42%, sys=19.14%, ctx=91520, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=109921KB/s, minb=109921KB/s, maxb=109921KB/s, mint=30500msec, maxt=30500msec
  WRITE: io=841688KB, aggrb=27596KB/s, minb=27596KB/s, maxb=27596KB/s, mint=30500msec, maxt=30500msec

Disk stats (read/write):
    dm-3: ios=547778/137138, merge=0/0, ticks=1112764/760488, in_queue=1873360, util=99.74%, aggrios=545238/137415, aggrmerge=5086/405, aggrticks=1107748/761360, aggrin_queue=1868864, aggrutil=99.63%
  sdp: ios=545238/137415, merge=5086/405, ticks=1107748/761360, in_queue=1868864, util=99.63%

kvm

• ext4 rw,relatime,nobarrier,data=ordered,errors=remount-ro 0 1

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [28797KB/6663KB/0KB /s] [6617/1530/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=686: Tue Apr 12 10:12:28 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=19017KB/s, iops=3120, runt=176293msec
    slat (usec): min=1, max=93960, avg=53.38, stdev=181.43
    clat (usec): min=111, max=1536.1K, avg=16315.84, stdev=22454.79
     lat (usec): min=433, max=1536.1K, avg=16372.43, stdev=22449.84
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11200], 50.00th=[13888], 60.00th=[16512],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49408], 99.50th=[82432], 99.90th=[329728], 99.95th=[460800],
     | 99.99th=[749568]
    bw (KB  /s): min=  555, max=29968, per=100.00%, avg=19030.43, stdev=5338.76
  write: io=841688KB, bw=4774.4KB/s, iops=780, runt=176293msec
    slat (usec): min=2, max=29676, avg=67.29, stdev=300.20
    clat (usec): min=179, max=1534.6K, avg=16353.61, stdev=22471.60
     lat (usec): min=452, max=1534.6K, avg=16424.05, stdev=22467.04
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11328], 50.00th=[13888], 60.00th=[16768],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49920], 99.50th=[84480], 99.90th=[321536], 99.95th=[444416],
     | 99.99th=[741376]
    bw (KB  /s): min=   42, max= 7746, per=100.00%, avg=4778.38, stdev=1386.59
    lat (usec) : 250=0.01%, 500=0.05%, 750=0.32%, 1000=0.74%
    lat (msec) : 2=3.53%, 4=7.44%, 10=23.16%, 20=35.92%, 50=27.86%
    lat (msec) : 100=0.59%, 250=0.23%, 500=0.12%, 750=0.03%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=3.12%, sys=48.14%, ctx=319451, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=19017KB/s, minb=19017KB/s, maxb=19017KB/s, mint=176293msec, maxt=176293msec
  WRITE: io=841688KB, aggrb=4774KB/s, minb=4774KB/s, maxb=4774KB/s, mint=176293msec, maxt=176293msec

Disk stats (read/write):
  sda: ios=539868/137353, merge=10782/802, ticks=8688496/2847176, in_queue=11534452, util=100.00%

• default ext4 errors=remount-ro 0 1

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [22426KB/5431KB/0KB /s] [5174/1254/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=707: Tue Apr 12 10:04:43 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=27987KB/s, iops=4592, runt=119793msec
    slat (usec): min=2, max=8121, avg=138.38, stdev=78.30
    clat (usec): min=61, max=878540, avg=10981.79, stdev=26616.61
     lat (usec): min=205, max=878545, avg=11122.39, stdev=26610.60
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   19], 99.90th=[  627], 99.95th=[  693],
     | 99.99th=[  848]
    bw (KB  /s): min=  419, max=56905, per=100.00%, avg=29243.35, stdev=10428.05
  write: io=841688KB, bw=7026.2KB/s, iops=1149, runt=119793msec
    slat (usec): min=3, max=8183, avg=149.49, stdev=114.76
    clat (usec): min=564, max=878335, avg=11044.70, stdev=27526.77
     lat (usec): min=608, max=878344, avg=11196.21, stdev=27539.90
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   20], 99.90th=[  635], 99.95th=[  693],
     | 99.99th=[  873]
    bw (KB  /s): min=   69, max=14976, per=100.00%, avg=7341.69, stdev=2672.17
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.06%, 4=0.32%, 10=76.18%, 20=23.02%, 50=0.19%
    lat (msec) : 100=0.02%, 250=0.02%, 500=0.04%, 750=0.13%, 1000=0.02%
  cpu          : usr=2.64%, sys=81.63%, ctx=480152, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=27986KB/s, minb=27986KB/s, maxb=27986KB/s, mint=119793msec, maxt=119793msec
  WRITE: io=841688KB, aggrb=7026KB/s, minb=7026KB/s, maxb=7026KB/s, mint=119793msec, maxt=119793msec

Disk stats (read/write):
  sda: ios=548732/137547, merge=838/94, ticks=976940/269344, in_queue=1243892, util=95.91%