Iscsi/tests: Difference between revisions
Jump to navigation
Jump to search
Bread-baker (talk | contribs) |
|||
| (One intermediate revision by one other user not shown) | |||
| Line 376: | Line 376: | ||
</pre> | </pre> | ||
== 2 drive sata mirror== | |||
with zil and cache on a intel super ssd [model later if needed, 200GB recommended model ] | |||
Latest revision as of 11:53, 29 November 2016
fio
for more info on fio see: TBD
apt-get install fio
create /root/fio.cfg :
echo "# from mir pve forum [global] description=Emulation of Intel IOmeter File Server Access Pattern [iometer] bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 rw=randrw rwmixread=80 direct=1 size=4g ioengine=libaio # IOMeter defines the server loads as the following: # iodepth=1 Linear # iodepth=4 Very Light # iodepth=8 Light # iodepth=64 Moderate # iodepth=256 Heavy iodepth=64" > /root/fio.cfg
mir's tests
- without nobarrier
/dev/sdb1 on /media/disk type ext4 (rw,relatime,data=ordered)
Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=18596KB/s, minb=18,596KB/s, maxb=18,596KB/s, mint=18,0281msec, maxt=18,0281msec WRITE: io=841688KB, aggrb=4668KB/s, minb=4,668KB/s, maxb=4,668KB/s, mint=18,0281msec, maxt=180281msec Disk stats (read/write): sdb: ios=549723/137605, merge=1/36, ticks=9265908/2204204, in_queue=11639448, util=100.00%
- nobarrier
/dev/sdb1 on /media/disk type ext4 (rw,relatime,nobarrier,data=ordered)
Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=30,526KB/s, minb=30,526KB/s, maxb=30,526KB/s, mint=10,9826msec, maxt=10,9826msec WRITE: io=841688KB, aggrb=7,663KB/s, minb=7,663KB/s, maxb=7,663KB/s, mint=10,9826msec, maxt=10,9826msec Disk stats (read/write): sdb: ios=550055/137665, merge=13/24, ticks=5510488/1558668, in_queue=7071272, util=100.00%
using XFS as filesystem inside the VM.
/dev/sdb1 on /media/disk type xfs (rw,relatime,attr2,inode64,noquota)
Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=38,831KB/s, minb=38,831KB/s, maxb=38,831KB/s, mint=86,337msec, maxt=86,337msec WRITE: io=841688KB, aggrb=9,748KB/s, minb=9,748KB/s, maxb=9,748KB/s, mint=86337msec, maxt=86337msec Disk stats (read/write): sdb: ios=549888/137583, merge=0/3, ticks=375584/77740, in_queue=452892, util=94.83%
sys4 tests
napp-it/omnios , motherboard: supermicro x9xcl-f , memory 28GB , cpu: Xeon(R) CPU E3-1280 V2 @ 3.60GHz [htop shows 8 cores ]
two SAS2008 IT mode HBA cards
this testing system had multipath issues. will try to redo tests on final system.
the following tests use a debian jessie virtual machines.
raidz1 with a log drive
zfs: raidz1 : 5 intel ssd pro series 2500 480GB + zil intel ssd s3700
lxc
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64 fio-2.1.11 Starting 1 process iometer: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 1 (f=1): [m(1)] [100.0% done] [252.7MB/65052KB/0KB /s] [59.3K/14.8K/0 iops] [eta 00m:00s] iometer: (groupid=0, jobs=1): err= 0: pid=2513: Sat Apr 9 15:24:14 2016 Description : [Emulation of Intel IOmeter File Server Access Pattern] read : io=3274.5MB, bw=303130KB/s, iops=49742, runt= 11060msec slat (usec): min=2, max=588, avg= 7.45, stdev=18.05 clat (usec): min=160, max=252974, avg=885.66, stdev=2765.50 lat (usec): min=167, max=252977, avg=893.38, stdev=2765.43 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 406], 10.00th=[ 482], 20.00th=[ 580], | 30.00th=[ 652], 40.00th=[ 716], 50.00th=[ 772], 60.00th=[ 836], | 70.00th=[ 892], 80.00th=[ 980], 90.00th=[ 1128], 95.00th=[ 1304], | 99.00th=[ 1832], 99.50th=[ 2320], 99.90th=[20608], 99.95th=[38656], | 99.99th=[136192] bw (KB /s): min=118739, max=444443, per=100.00%, avg=305665.86, stdev=75220.28 write: io=841688KB, bw=76102KB/s, iops=12445, runt= 11060msec slat (usec): min=3, max=1020, avg= 9.17, stdev=20.35 clat (usec): min=395, max=264907, avg=1552.83, stdev=5218.00 lat (usec): min=402, max=264915, avg=1562.29, stdev=5217.88 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 804], 10.00th=[ 908], 20.00th=[ 1032], | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384], | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1864], 95.00th=[ 2128], | 99.00th=[ 2960], 99.50th=[ 4320], 99.90th=[61696], 99.95th=[121344], | 99.99th=[252928] bw (KB /s): min=28575, max=111024, per=100.00%, avg=76804.24, stdev=18922.29 lat (usec) : 250=0.01%, 500=9.42%, 750=27.66%, 1000=31.51% lat (msec) : 2=29.46%, 4=1.58%, 10=0.12%, 20=0.06%, 50=0.12% lat (msec) : 100=0.02%, 250=0.03%, 500=0.01% cpu : usr=12.19%, sys=56.86%, ctx=17722, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=303130KB/s, minb=303130KB/s, maxb=303130KB/s, mint=11060msec, maxt=11060msec WRITE: io=841688KB, aggrb=76101KB/s, minb=76101KB/s, maxb=76101KB/s, mint=11060msec, maxt=11060msec Disk stats (read/write): dm-2: ios=543761/136160, merge=0/0, ticks=349496/178408, in_queue=527996, util=99.14%, aggrios=547260/137478, aggrmerge=2955/219, aggrticks=351272/179892, aggrin_queue=530992, aggrutil=98.92% sdk: ios=547260/137478, merge=2955/219, ticks=351272/179892, in_queue=530992, util=98.92%
kvm
/dev/sda1 on / type ext4 (rw,relatime,nobarrier,data=ordered)
at pve cache set to default nocache.
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64 fio-2.1.11 Starting 1 process Jobs: 1 (f=1): [m(1)] [100.0% done] [13251KB/3173KB/0KB /s] [3093/735/0 iops] [eta 00m:00s] iometer: (groupid=0, jobs=1): err= 0: pid=695: Sat Apr 9 19:12:16 2016 Description : [Emulation of Intel IOmeter File Server Access Pattern] read : io=3274.5MB, bw=13694KB/s, iops=2247, runt=244825msec slat (usec): min=1, max=34337, avg=10.52, stdev=160.01 clat (usec): min=96, max=485692, avg=22701.29, stdev=14578.86 lat (usec): min=660, max=485701, avg=22712.10, stdev=14578.48 clat percentiles (usec): | 1.00th=[ 1112], 5.00th=[ 2800], 10.00th=[ 4960], 20.00th=[ 9152], | 30.00th=[13504], 40.00th=[17792], 50.00th=[22144], 60.00th=[26496], | 70.00th=[30848], 80.00th=[35072], 90.00th=[40192], 95.00th=[43264], | 99.00th=[52480], 99.50th=[60160], 99.90th=[85504], 99.95th=[113152], | 99.99th=[296960] bw (KB /s): min= 5004, max=25066, per=100.00%, avg=13713.01, stdev=3454.14 write: io=841688KB, bw=3437.1KB/s, iops=562, runt=244825msec slat (usec): min=3, max=32123, avg=17.92, stdev=409.82 clat (usec): min=868, max=483835, avg=23016.88, stdev=14330.54 lat (usec): min=881, max=483848, avg=23035.12, stdev=14335.88 clat percentiles (usec): | 1.00th=[ 1416], 5.00th=[ 3056], 10.00th=[ 5216], 20.00th=[ 9536], | 30.00th=[13888], 40.00th=[18304], 50.00th=[22400], 60.00th=[26752], | 70.00th=[31104], 80.00th=[35584], 90.00th=[40192], 95.00th=[43776], | 99.00th=[53504], 99.50th=[61184], 99.90th=[85504], 99.95th=[98816], | 99.99th=[264192] bw (KB /s): min= 1422, max= 6579, per=100.00%, avg=3443.28, stdev=926.27 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.51% lat (msec) : 2=2.47%, 4=4.68%, 10=13.98%, 20=23.24%, 50=53.71% lat (msec) : 100=1.32%, 250=0.04%, 500=0.02% cpu : usr=10.73%, sys=23.07%, ctx=673631, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: io=3274.5MB, aggrb=13693KB/s, minb=13693KB/s, maxb=13693KB/s, mint=244825msec, maxt=244825msec WRITE: io=841688KB, aggrb=3437KB/s, minb=3437KB/s, maxb=3437KB/s, mint=244825msec, maxt=244825msec Disk stats (read/write): sda: ios=539489/137206, merge=11367/712, ticks=12209328/3240400, in_queue=15451416, util=100.00%
4 drive mirror, without a log drive
four intel ssd pro series 2500 480GB attached to its own HBA.
lxc
3032-jessie ~ # fio /root/fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [93189KB/23350KB/0KB /s] [21.7K/5469/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1857: Tue Apr 12 09:39:34 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=109922KB/s, iops=18037, runt= 30500msec
slat (usec): min=2, max=8635, avg= 6.00, stdev=16.12
clat (usec): min=144, max=43100, avg=2123.05, stdev=2388.69
lat (usec): min=166, max=43103, avg=2129.27, stdev=2388.60
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 418], 10.00th=[ 486], 20.00th=[ 604],
| 30.00th=[ 732], 40.00th=[ 892], 50.00th=[ 1160], 60.00th=[ 1624],
| 70.00th=[ 2384], 80.00th=[ 3312], 90.00th=[ 4896], 95.00th=[ 6496],
| 99.00th=[10816], 99.50th=[13248], 99.90th=[22144], 99.95th=[24192],
| 99.99th=[31616]
bw (KB /s): min=79049, max=218569, per=100.00%, avg=110088.87, stdev=30276.89
write: io=841688KB, bw=27596KB/s, iops=4512, runt= 30500msec
slat (usec): min=3, max=8921, avg= 7.78, stdev=35.57
clat (usec): min=462, max=54371, avg=5654.28, stdev=3658.82
lat (usec): min=469, max=54379, avg=5662.30, stdev=3658.89
clat percentiles (usec):
| 1.00th=[ 1032], 5.00th=[ 1448], 10.00th=[ 1880], 20.00th=[ 2704],
| 30.00th=[ 3408], 40.00th=[ 4128], 50.00th=[ 4896], 60.00th=[ 5728],
| 70.00th=[ 6752], 80.00th=[ 8096], 90.00th=[10304], 95.00th=[12352],
| 99.00th=[17536], 99.50th=[21632], 99.90th=[28288], 99.95th=[31104],
| 99.99th=[41216]
bw (KB /s): min=19175, max=53703, per=100.00%, avg=27661.45, stdev=7629.12
lat (usec) : 250=0.09%, 500=8.83%, 750=16.25%, 1000=10.74%
lat (msec) : 2=18.74%, 4=21.13%, 10=20.97%, 20=2.96%, 50=0.31%
lat (msec) : 100=0.01%
cpu : usr=5.42%, sys=19.14%, ctx=91520, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=109921KB/s, minb=109921KB/s, maxb=109921KB/s, mint=30500msec, maxt=30500msec
WRITE: io=841688KB, aggrb=27596KB/s, minb=27596KB/s, maxb=27596KB/s, mint=30500msec, maxt=30500msec
Disk stats (read/write):
dm-3: ios=547778/137138, merge=0/0, ticks=1112764/760488, in_queue=1873360, util=99.74%, aggrios=545238/137415, aggrmerge=5086/405, aggrticks=1107748/761360, aggrin_queue=1868864, aggrutil=99.63%
sdp: ios=545238/137415, merge=5086/405, ticks=1107748/761360, in_queue=1868864, util=99.63%
kvm
- ext4 rw,relatime,nobarrier,data=ordered,errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [28797KB/6663KB/0KB /s] [6617/1530/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=686: Tue Apr 12 10:12:28 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=19017KB/s, iops=3120, runt=176293msec
slat (usec): min=1, max=93960, avg=53.38, stdev=181.43
clat (usec): min=111, max=1536.1K, avg=16315.84, stdev=22454.79
lat (usec): min=433, max=1536.1K, avg=16372.43, stdev=22449.84
clat percentiles (usec):
| 1.00th=[ 964], 5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
| 30.00th=[ 8640], 40.00th=[11200], 50.00th=[13888], 60.00th=[16512],
| 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
| 99.00th=[49408], 99.50th=[82432], 99.90th=[329728], 99.95th=[460800],
| 99.99th=[749568]
bw (KB /s): min= 555, max=29968, per=100.00%, avg=19030.43, stdev=5338.76
write: io=841688KB, bw=4774.4KB/s, iops=780, runt=176293msec
slat (usec): min=2, max=29676, avg=67.29, stdev=300.20
clat (usec): min=179, max=1534.6K, avg=16353.61, stdev=22471.60
lat (usec): min=452, max=1534.6K, avg=16424.05, stdev=22467.04
clat percentiles (usec):
| 1.00th=[ 956], 5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
| 30.00th=[ 8640], 40.00th=[11328], 50.00th=[13888], 60.00th=[16768],
| 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
| 99.00th=[49920], 99.50th=[84480], 99.90th=[321536], 99.95th=[444416],
| 99.99th=[741376]
bw (KB /s): min= 42, max= 7746, per=100.00%, avg=4778.38, stdev=1386.59
lat (usec) : 250=0.01%, 500=0.05%, 750=0.32%, 1000=0.74%
lat (msec) : 2=3.53%, 4=7.44%, 10=23.16%, 20=35.92%, 50=27.86%
lat (msec) : 100=0.59%, 250=0.23%, 500=0.12%, 750=0.03%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=3.12%, sys=48.14%, ctx=319451, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=19017KB/s, minb=19017KB/s, maxb=19017KB/s, mint=176293msec, maxt=176293msec
WRITE: io=841688KB, aggrb=4774KB/s, minb=4774KB/s, maxb=4774KB/s, mint=176293msec, maxt=176293msec
Disk stats (read/write):
sda: ios=539868/137353, merge=10782/802, ticks=8688496/2847176, in_queue=11534452, util=100.00%
- default ext4 errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [22426KB/5431KB/0KB /s] [5174/1254/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=707: Tue Apr 12 10:04:43 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=27987KB/s, iops=4592, runt=119793msec
slat (usec): min=2, max=8121, avg=138.38, stdev=78.30
clat (usec): min=61, max=878540, avg=10981.79, stdev=26616.61
lat (usec): min=205, max=878545, avg=11122.39, stdev=26610.60
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
| 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 17], 99.50th=[ 19], 99.90th=[ 627], 99.95th=[ 693],
| 99.99th=[ 848]
bw (KB /s): min= 419, max=56905, per=100.00%, avg=29243.35, stdev=10428.05
write: io=841688KB, bw=7026.2KB/s, iops=1149, runt=119793msec
slat (usec): min=3, max=8183, avg=149.49, stdev=114.76
clat (usec): min=564, max=878335, avg=11044.70, stdev=27526.77
lat (usec): min=608, max=878344, avg=11196.21, stdev=27539.90
clat percentiles (msec):
| 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10],
| 30.00th=[ 10], 40.00th=[ 10], 50.00th=[ 10], 60.00th=[ 10],
| 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
| 99.00th=[ 17], 99.50th=[ 20], 99.90th=[ 635], 99.95th=[ 693],
| 99.99th=[ 873]
bw (KB /s): min= 69, max=14976, per=100.00%, avg=7341.69, stdev=2672.17
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.06%, 4=0.32%, 10=76.18%, 20=23.02%, 50=0.19%
lat (msec) : 100=0.02%, 250=0.02%, 500=0.04%, 750=0.13%, 1000=0.02%
cpu : usr=2.64%, sys=81.63%, ctx=480152, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=27986KB/s, minb=27986KB/s, maxb=27986KB/s, mint=119793msec, maxt=119793msec
WRITE: io=841688KB, aggrb=7026KB/s, minb=7026KB/s, maxb=7026KB/s, mint=119793msec, maxt=119793msec
Disk stats (read/write):
sda: ios=548732/137547, merge=838/94, ticks=976940/269344, in_queue=1243892, util=95.91%
6 sata drive mirror with ssd log
I scrapped this set up. and went with a 2 drive mirror with the newest disks [ some of the others were 4-5 years old ].
kvm
jessie-template ~ # fio /fbc/adm/fio.cfg # fio.cfg contains mir's code above . TBD upload fio.cfg or copy paste to screen
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [119.1MB/31420KB/0KB /s] [28.5K/7151/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1369: Mon Jun 6 12:49:29 2016
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=3274.5MB, bw=164191KB/s, iops=26943, runt= 20419msec
slat (usec): min=1, max=1030, avg= 6.10, stdev= 4.59
clat (usec): min=117, max=166663, avg=1716.19, stdev=4230.12
lat (usec): min=126, max=166670, avg=1722.55, stdev=4230.16
clat percentiles (usec):
| 1.00th=[ 524], 5.00th=[ 876], 10.00th=[ 940], 20.00th=[ 996],
| 30.00th=[ 1032], 40.00th=[ 1064], 50.00th=[ 1096], 60.00th=[ 1144],
| 70.00th=[ 1192], 80.00th=[ 1272], 90.00th=[ 1624], 95.00th=[ 2160],
| 99.00th=[21120], 99.50th=[37632], 99.90th=[55040], 99.95th=[57600],
| 99.99th=[62720]
bw (KB /s): min=95414, max=313106, per=100.00%, avg=164904.00, stdev=49364.41
write: io=841688KB, bw=41221KB/s, iops=6740, runt= 20419msec
slat (usec): min=2, max=44634, avg= 8.04, stdev=137.58
clat (usec): min=419, max=89028, avg=2592.06, stdev=5228.52
lat (usec): min=526, max=89034, avg=2600.37, stdev=5230.32
clat percentiles (usec):
| 1.00th=[ 1096], 5.00th=[ 1240], 10.00th=[ 1304], 20.00th=[ 1368],
| 30.00th=[ 1416], 40.00th=[ 1464], 50.00th=[ 1512], 60.00th=[ 1576],
| 70.00th=[ 1656], 80.00th=[ 1816], 90.00th=[ 2448], 95.00th=[ 4896],
| 99.00th=[30848], 99.50th=[41216], 99.90th=[56576], 99.95th=[59136],
| 99.99th=[61696]
bw (KB /s): min=23379, max=78212, per=100.00%, avg=41429.42, stdev=12385.69
lat (usec) : 250=0.11%, 500=0.62%, 750=1.42%, 1000=14.84%
lat (msec) : 2=74.72%, 4=5.20%, 10=0.72%, 20=1.05%, 50=1.12%
lat (msec) : 100=0.21%, 250=0.01%
cpu : usr=11.13%, sys=25.59%, ctx=304291, majf=0, minf=8
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=164191KB/s, minb=164191KB/s, maxb=164191KB/s, mint=20419msec, maxt=20419msec
WRITE: io=841688KB, aggrb=41220KB/s, minb=41220KB/s, maxb=41220KB/s, mint=20419msec, maxt=20419msec
Disk stats (read/write):
vda: ios=548282/137250, merge=0/0, ticks=934124/351920, in_queue=1286136, util=99.50%
2 drive sata mirror
with zil and cache on a intel super ssd [model later if needed, 200GB recommended model ]