Iscsi/tests
fio
for more info on fio see: TBD
apt-get install fio
create /root/fio.cfg :
echo "# from mir pve forum [global] description=Emulation of Intel IOmeter File Server Access Pattern [iometer] bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 rw=randrw rwmixread=80 direct=1 size=4g ioengine=libaio # IOMeter defines the server loads as the following: # iodepth=1 Linear # iodepth=4 Very Light # iodepth=8 Light # iodepth=64 Moderate # iodepth=256 Heavy iodepth=64" > /root/fio.cfg
sys4 tests
napp-it/omnios , motherboard: supermicro x9xcl-f , memory 28GB , cpu:
two SAS2008 IT mode HBA cards
the following tests use a debian jessie virtual machines.
raidz1 with a log drive
zfs: raidz1 : 5 intel ssd pro series 2500 480GB + zil intel ssd s3700
TBD copy and paste from forum thread.
lxc
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64 fio-2.1.11 Starting 1 process iometer: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m(1)] [100.0% done] [252.7MB/65052KB/0KB /s] [59.3K/14.8K/0 iops] [eta 00m:00s] iometer: (groupid=0, jobs=1): err= 0: pid=2513: Sat Apr 9 15:24:14 2016 Description : [Emulation of Intel IOmeter File Server Access Pattern] read : io=3274.5MB, bw=303130KB/s, iops=49742, runt= 11060msec slat (usec): min=2, max=588, avg= 7.45, stdev=18.05 clat (usec): min=160, max=252974, avg=885.66, stdev=2765.50 lat (usec): min=167, max=252977, avg=893.38, stdev=2765.43 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 406], 10.00th=[ 482], 20.00th=[ 580], | 30.00th=[ 652], 40.00th=[ 716], 50.00th=[ 772], 60.00th=[ 836], | 70.00th=[ 892], 80.00th=[ 980], 90.00th=[ 1128], 95.00th=[ 1304], | 99.00th=[ 1832], 99.50th=[ 2320], 99.90th=[20608], 99.95th=[38656], | 99.99th=[136192] bw (KB /s): min=118739, max=444443, per=100.00%, avg=305665.86, stdev=75220.28 write: io=841688KB, bw=76102KB/s, iops=12445, runt= 11060msec slat (usec): min=3, max=1020, avg= 9.17, stdev=20.35 clat (usec): min=395, max=264907, avg=1552.83, stdev=5218.00 lat (usec): min=402, max=264915, avg=1562.29, stdev=5217.88 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 804], 10.00th=[ 908], 20.00th=[ 1032], | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384], | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1864], 95.00th=[ 2128], | 99.00th=[ 2960], 99.50th=[ 4320], 99.90th=[61696], 99.95th=[121344], | 99.99th=[252928] bw (KB /s): min=28575, max=111024, per=100.00%, avg=76804.24, stdev=18922.29 lat (usec) : 250=0.01%, 500=9.42%, 750=27.66%, 1000=31.51% lat (msec) : 2=29.46%, 4=1.58%, 10=0.12%, 20=0.06%, 50=0.12% lat (msec) : 100=0.02%, 250=0.03%, 500=0.01% cpu : usr=12.19%, sys=56.86%, ctx=17722, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=303130KB/s, minb=303130KB/s, maxb=303130KB/s, mint=11060msec, maxt=11060msec WRITE: io=841688KB, aggrb=76101KB/s, minb=76101KB/s, maxb=76101KB/s, mint=11060msec, maxt=11060msec Disk stats (read/write): dm-2: ios=543761/136160, merge=0/0, ticks=349496/178408, in_queue=527996, util=99.14%, aggrios=547260/137478, aggrmerge=2955/219, aggrticks=351272/179892, aggrin_queue=530992, aggrutil=98.92% sdk: ios=547260/137478, merge=2955/219, ticks=351272/179892, in_queue=530992, util=98.92%
kvm
4 drive mirror, without a log drive
4 intel ssd model ___ , no zil
lxc
3032-jessie ~ # fio /root/fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [93189KB/23350KB/0KB /s] [21.7K/5469/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1857: Tue Apr 12 09:39:34 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=109922KB/s, iops=18037, runt= 30500msec
slat (usec): min=2, max=8635, avg= 6.00, stdev=16.12
clat (usec): min=144, max=43100, avg=2123.05, stdev=2388.69
lat (usec): min=166, max=43103, avg=2129.27, stdev=2388.60
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 418], 10.00th=[ 486], 20.00th=[ 604],
| 30.00th=[ 732], 40.00th=[ 892], 50.00th=[ 1160], 60.00th=[ 1624],
| 70.00th=[ 2384], 80.00th=[ 3312], 90.00th=[ 4896], 95.00th=[ 6496],
| 99.00th=[10816], 99.50th=[13248], 99.90th=[22144], 99.95th=[24192],
| 99.99th=[31616]
bw (KB /s): min=79049, max=218569, per=100.00%, avg=110088.87, stdev=30276.89
write: io=841688KB, bw=27596KB/s, iops=4512, runt= 30500msec
slat (usec): min=3, max=8921, avg= 7.78, stdev=35.57
clat (usec): min=462, max=54371, avg=5654.28, stdev=3658.82
lat (usec): min=469, max=54379, avg=5662.30, stdev=3658.89
clat percentiles (usec):
| 1.00th=[ 1032], 5.00th=[ 1448], 10.00th=[ 1880], 20.00th=[ 2704],
| 30.00th=[ 3408], 40.00th=[ 4128], 50.00th=[ 4896], 60.00th=[ 5728],
| 70.00th=[ 6752], 80.00th=[ 8096], 90.00th=[10304], 95.00th=[12352],
| 99.00th=[17536], 99.50th=[21632], 99.90th=[28288], 99.95th=[31104],
| 99.99th=[41216]
bw (KB /s): min=19175, max=53703, per=100.00%, avg=27661.45, stdev=7629.12
lat (usec) : 250=0.09%, 500=8.83%, 750=16.25%, 1000=10.74%
lat (msec) : 2=18.74%, 4=21.13%, 10=20.97%, 20=2.96%, 50=0.31%
lat (msec) : 100=0.01%
cpu : usr=5.42%, sys=19.14%, ctx=91520, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=109921KB/s, minb=109921KB/s, maxb=109921KB/s, mint=30500msec, maxt=30500msec
WRITE: io=841688KB, aggrb=27596KB/s, minb=27596KB/s, maxb=27596KB/s, mint=30500msec, maxt=30500msec
Disk stats (read/write):
dm-3: ios=547778/137138, merge=0/0, ticks=1112764/760488, in_queue=1873360, util=99.74%, aggrios=545238/137415, aggrmerge=5086/405, aggrticks=1107748/761360, aggrin_queue=1868864, aggrutil=99.63%
sdp: ios=545238/137415, merge=5086/405, ticks=1107748/761360, in_queue=1868864, util=99.63%
kvm
- ext4 rw,relatime,nobarrier,data=ordered,errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [28797KB/6663KB/0KB /s] [6617/1530/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=686: Tue Apr 12 10:12:28 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=19017KB/s, iops=3120, runt=176293msec
slat (usec): min=1, max=93960, avg=53.38, stdev=181.43
clat (usec): min=111, max=1536.1K, avg=16315.84, stdev=22454.79
lat (usec): min=433, max=1536.1K, avg=16372.43, stdev=22449.84
clat percentiles (usec):
| 1.00th=[ 964], 5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
| 30.00th=[ 8640], 40.00th=[11200], 50.00th=[13888], 60.00th=[16512],
| 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
| 99.00th=[49408], 99.50th=[82432], 99.90th=[329728], 99.95th=[460800],
| 99.99th=[749568]
bw (KB /s): min= 555, max=29968, per=100.00%, avg=19030.43, stdev=5338.76
write: io=841688KB, bw=4774.4KB/s, iops=780, runt=176293msec
slat (usec): min=2, max=29676, avg=67.29, stdev=300.20
clat (usec): min=179, max=1534.6K, avg=16353.61, stdev=22471.60
lat (usec): min=452, max=1534.6K, avg=16424.05, stdev=22467.04
clat percentiles (usec):
| 1.00th=[ 956], 5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
| 30.00th=[ 8640], 40.00th=[11328], 50.00th=[13888], 60.00th=[16768],
| 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
| 99.00th=[49920], 99.50th=[84480], 99.90th=[321536], 99.95th=[444416],
| 99.99th=[741376]
bw (KB /s): min= 42, max= 7746, per=100.00%, avg=4778.38, stdev=1386.59
lat (usec) : 250=0.01%, 500=0.05%, 750=0.32%, 1000=0.74%
lat (msec) : 2=3.53%, 4=7.44%, 10=23.16%, 20=35.92%, 50=27.86%
lat (msec) : 100=0.59%, 250=0.23%, 500=0.12%, 750=0.03%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=3.12%, sys=48.14%, ctx=319451, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=19017KB/s, minb=19017KB/s, maxb=19017KB/s, mint=176293msec, maxt=176293msec
WRITE: io=841688KB, aggrb=4774KB/s, minb=4774KB/s, maxb=4774KB/s, mint=176293msec, maxt=176293msec
Disk stats (read/write):
sda: ios=539868/137353, merge=10782/802, ticks=8688496/2847176, in_queue=11534452, util=100.00%
- default ext4 errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [22426KB/5431KB/0KB /s] [5174/1254/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=707: Tue Apr 12 10:04:43 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=27987KB/s, iops=4592, runt=119793msec
slat (usec): min=2, max=8121, avg=138.38, stdev=78.30
clat (usec): min=61, max=878540, avg=10981.79, stdev=26616.61
lat (usec): min=205, max=878545, avg=11122.39, stdev=26610.60
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
| 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 17], 99.50th=[ 19], 99.90th=[ 627], 99.95th=[ 693],
| 99.99th=[ 848]
bw (KB /s): min= 419, max=56905, per=100.00%, avg=29243.35, stdev=10428.05
write: io=841688KB, bw=7026.2KB/s, iops=1149, runt=119793msec
slat (usec): min=3, max=8183, avg=149.49, stdev=114.76
clat (usec): min=564, max=878335, avg=11044.70, stdev=27526.77
lat (usec): min=608, max=878344, avg=11196.21, stdev=27539.90
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
| 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 635], 99.95th=[ 693],
| 99.99th=[ 873]
bw (KB /s): min= 69, max=14976, per=100.00%, avg=7341.69, stdev=2672.17
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.06%, 4=0.32%, 10=76.18%, 20=23.02%, 50=0.19%
lat (msec) : 100=0.02%, 250=0.02%, 500=0.04%, 750=0.13%, 1000=0.02%
cpu : usr=2.64%, sys=81.63%, ctx=480152, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=27986KB/s, minb=27986KB/s, maxb=27986KB/s, mint=119793msec, maxt=119793msec
WRITE: io=841688KB, aggrb=7026KB/s, minb=7026KB/s, maxb=7026KB/s, mint=119793msec, maxt=119793msec
Disk stats (read/write):
sda: ios=548732/137547, merge=838/94, ticks=976940/269344, in_queue=1243892, util=95.91%