Iscsi/tests

fio

for more info on fio see: TBD

apt-get install fio

create /root/fio.cfg :

echo "# from mir pve forum
[global]
description=Emulation of Intel IOmeter File Server Access Pattern

[iometer]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=80
direct=1
size=4g
ioengine=libaio
# IOMeter defines the server loads as the following:
# iodepth=1    Linear
# iodepth=4    Very Light
# iodepth=8    Light
# iodepth=64    Moderate
# iodepth=256    Heavy
iodepth=64" > /root/fio.cfg

mir's tests

• without nobarrier

/dev/sdb1 on /media/disk type ext4 (rw,relatime,data=ordered)

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=18596KB/s, minb=18,596KB/s, maxb=18,596KB/s, mint=18,0281msec, maxt=18,0281msec
WRITE: io=841688KB, aggrb=4668KB/s, minb=4,668KB/s, maxb=4,668KB/s, mint=18,0281msec, maxt=180281msec

Disk stats (read/write):
sdb: ios=549723/137605, merge=1/36, ticks=9265908/2204204, in_queue=11639448, util=100.00%

• nobarrier

/dev/sdb1 on /media/disk type ext4 (rw,relatime,nobarrier,data=ordered)

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=30,526KB/s, minb=30,526KB/s, maxb=30,526KB/s, mint=10,9826msec, maxt=10,9826msec
WRITE: io=841688KB, aggrb=7,663KB/s, minb=7,663KB/s, maxb=7,663KB/s, mint=10,9826msec, maxt=10,9826msec

Disk stats (read/write):
sdb: ios=550055/137665, merge=13/24, ticks=5510488/1558668, in_queue=7071272, util=100.00%

using XFS as filesystem inside the VM.

/dev/sdb1 on /media/disk type xfs (rw,relatime,attr2,inode64,noquota)

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=38,831KB/s, minb=38,831KB/s, maxb=38,831KB/s, mint=86,337msec, maxt=86,337msec
WRITE: io=841688KB, aggrb=9,748KB/s, minb=9,748KB/s, maxb=9,748KB/s, mint=86337msec, maxt=86337msec

Disk stats (read/write):
sdb: ios=549888/137583, merge=0/3, ticks=375584/77740, in_queue=452892, util=94.83%

sys4 tests

napp-it/omnios , motherboard: supermicro x9xcl-f , memory 28GB , cpu: Xeon(R) CPU E3-1280 V2 @ 3.60GHz [htop shows 8 cores ]

two SAS2008 IT mode HBA cards

this testing system had multipath issues. will try to redo tests on final system.

the following tests use a debian jessie virtual machines.

raidz1 with a log drive

zfs: raidz1 : 5 intel ssd pro series 2500 480GB + zil intel ssd s3700

lxc

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [252.7MB/65052KB/0KB /s] [59.3K/14.8K/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=2513: Sat Apr  9 15:24:14 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=303130KB/s, iops=49742, runt= 11060msec
  slat (usec): min=2, max=588, avg= 7.45, stdev=18.05
  clat (usec): min=160, max=252974, avg=885.66, stdev=2765.50
  lat (usec): min=167, max=252977, avg=893.38, stdev=2765.43
  clat percentiles (usec):
  |  1.00th=[  334],  5.00th=[  406], 10.00th=[  482], 20.00th=[  580],
  | 30.00th=[  652], 40.00th=[  716], 50.00th=[  772], 60.00th=[  836],
  | 70.00th=[  892], 80.00th=[  980], 90.00th=[ 1128], 95.00th=[ 1304],
  | 99.00th=[ 1832], 99.50th=[ 2320], 99.90th=[20608], 99.95th=[38656],
  | 99.99th=[136192]
  bw (KB  /s): min=118739, max=444443, per=100.00%, avg=305665.86, stdev=75220.28
  write: io=841688KB, bw=76102KB/s, iops=12445, runt= 11060msec
  slat (usec): min=3, max=1020, avg= 9.17, stdev=20.35
  clat (usec): min=395, max=264907, avg=1552.83, stdev=5218.00
  lat (usec): min=402, max=264915, avg=1562.29, stdev=5217.88
  clat percentiles (usec):
  |  1.00th=[  644],  5.00th=[  804], 10.00th=[  908], 20.00th=[ 1032],
  | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384],
  | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1864], 95.00th=[ 2128],
  | 99.00th=[ 2960], 99.50th=[ 4320], 99.90th=[61696], 99.95th=[121344],
  | 99.99th=[252928]
  bw (KB  /s): min=28575, max=111024, per=100.00%, avg=76804.24, stdev=18922.29
  lat (usec) : 250=0.01%, 500=9.42%, 750=27.66%, 1000=31.51%
  lat (msec) : 2=29.46%, 4=1.58%, 10=0.12%, 20=0.06%, 50=0.12%
  lat (msec) : 100=0.02%, 250=0.03%, 500=0.01%
  cpu  : usr=12.19%, sys=56.86%, ctx=17722, majf=0, minf=8
  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
  submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
  issued  : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
  latency  : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: io=3274.5MB, aggrb=303130KB/s, minb=303130KB/s, maxb=303130KB/s, mint=11060msec, maxt=11060msec
  WRITE: io=841688KB, aggrb=76101KB/s, minb=76101KB/s, maxb=76101KB/s, mint=11060msec, maxt=11060msec

Disk stats (read/write):
  dm-2: ios=543761/136160, merge=0/0, ticks=349496/178408, in_queue=527996, util=99.14%, aggrios=547260/137478, aggrmerge=2955/219, aggrticks=351272/179892, aggrin_queue=530992, aggrutil=98.92%
  sdk: ios=547260/137478, merge=2955/219, ticks=351272/179892, in_queue=530992, util=98.92%

kvm

/dev/sda1 on / type ext4 (rw,relatime,nobarrier,data=ordered)

at pve cache set to default nocache.

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [13251KB/3173KB/0KB /s] [3093/735/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=695: Sat Apr  9 19:12:16 2016   
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]   
  read : io=3274.5MB, bw=13694KB/s, iops=2247, runt=244825msec   
  slat (usec): min=1, max=34337, avg=10.52, stdev=160.01
  clat (usec): min=96, max=485692, avg=22701.29, stdev=14578.86
  lat (usec): min=660, max=485701, avg=22712.10, stdev=14578.48
  clat percentiles (usec):
  |  1.00th=[ 1112],  5.00th=[ 2800], 10.00th=[ 4960], 20.00th=[ 9152],
  | 30.00th=[13504], 40.00th=[17792], 50.00th=[22144], 60.00th=[26496],
  | 70.00th=[30848], 80.00th=[35072], 90.00th=[40192], 95.00th=[43264],
  | 99.00th=[52480], 99.50th=[60160], 99.90th=[85504], 99.95th=[113152],
  | 99.99th=[296960]
  bw (KB  /s): min= 5004, max=25066, per=100.00%, avg=13713.01, stdev=3454.14
  write: io=841688KB, bw=3437.1KB/s, iops=562, runt=244825msec
  slat (usec): min=3, max=32123, avg=17.92, stdev=409.82
  clat (usec): min=868, max=483835, avg=23016.88, stdev=14330.54
  lat (usec): min=881, max=483848, avg=23035.12, stdev=14335.88
  clat percentiles (usec):
  |  1.00th=[ 1416],  5.00th=[ 3056], 10.00th=[ 5216], 20.00th=[ 9536],
  | 30.00th=[13888], 40.00th=[18304], 50.00th=[22400], 60.00th=[26752],
  | 70.00th=[31104], 80.00th=[35584], 90.00th=[40192], 95.00th=[43776],
  | 99.00th=[53504], 99.50th=[61184], 99.90th=[85504], 99.95th=[98816],
  | 99.99th=[264192]
  bw (KB  /s): min= 1422, max= 6579, per=100.00%, avg=3443.28, stdev=926.27
  lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.51%
  lat (msec) : 2=2.47%, 4=4.68%, 10=13.98%, 20=23.24%, 50=53.71%
  lat (msec) : 100=1.32%, 250=0.04%, 500=0.02%
  cpu  : usr=10.73%, sys=23.07%, ctx=673631, majf=0, minf=8
  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
  submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
  issued  : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
  latency  : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: io=3274.5MB, aggrb=13693KB/s, minb=13693KB/s, maxb=13693KB/s, mint=244825msec, maxt=244825msec
  WRITE: io=841688KB, aggrb=3437KB/s, minb=3437KB/s, maxb=3437KB/s, mint=244825msec, maxt=244825msec

Disk stats (read/write):
  sda: ios=539489/137206, merge=11367/712, ticks=12209328/3240400, in_queue=15451416, util=100.00%

4 drive mirror, without a log drive

four intel ssd pro series 2500 480GB attached to its own HBA.

lxc

3032-jessie  ~ # fio /root/fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [93189KB/23350KB/0KB /s] [21.7K/5469/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1857: Tue Apr 12 09:39:34 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=109922KB/s, iops=18037, runt= 30500msec
    slat (usec): min=2, max=8635, avg= 6.00, stdev=16.12
    clat (usec): min=144, max=43100, avg=2123.05, stdev=2388.69
     lat (usec): min=166, max=43103, avg=2129.27, stdev=2388.60
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  418], 10.00th=[  486], 20.00th=[  604],
     | 30.00th=[  732], 40.00th=[  892], 50.00th=[ 1160], 60.00th=[ 1624],
     | 70.00th=[ 2384], 80.00th=[ 3312], 90.00th=[ 4896], 95.00th=[ 6496],
     | 99.00th=[10816], 99.50th=[13248], 99.90th=[22144], 99.95th=[24192],
     | 99.99th=[31616]
    bw (KB  /s): min=79049, max=218569, per=100.00%, avg=110088.87, stdev=30276.89
  write: io=841688KB, bw=27596KB/s, iops=4512, runt= 30500msec
    slat (usec): min=3, max=8921, avg= 7.78, stdev=35.57
    clat (usec): min=462, max=54371, avg=5654.28, stdev=3658.82
     lat (usec): min=469, max=54379, avg=5662.30, stdev=3658.89
    clat percentiles (usec):
     |  1.00th=[ 1032],  5.00th=[ 1448], 10.00th=[ 1880], 20.00th=[ 2704],
     | 30.00th=[ 3408], 40.00th=[ 4128], 50.00th=[ 4896], 60.00th=[ 5728],
     | 70.00th=[ 6752], 80.00th=[ 8096], 90.00th=[10304], 95.00th=[12352],
     | 99.00th=[17536], 99.50th=[21632], 99.90th=[28288], 99.95th=[31104],
     | 99.99th=[41216]
    bw (KB  /s): min=19175, max=53703, per=100.00%, avg=27661.45, stdev=7629.12
    lat (usec) : 250=0.09%, 500=8.83%, 750=16.25%, 1000=10.74%
    lat (msec) : 2=18.74%, 4=21.13%, 10=20.97%, 20=2.96%, 50=0.31%
    lat (msec) : 100=0.01%
  cpu          : usr=5.42%, sys=19.14%, ctx=91520, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=109921KB/s, minb=109921KB/s, maxb=109921KB/s, mint=30500msec, maxt=30500msec
  WRITE: io=841688KB, aggrb=27596KB/s, minb=27596KB/s, maxb=27596KB/s, mint=30500msec, maxt=30500msec

Disk stats (read/write):
    dm-3: ios=547778/137138, merge=0/0, ticks=1112764/760488, in_queue=1873360, util=99.74%, aggrios=545238/137415, aggrmerge=5086/405, aggrticks=1107748/761360, aggrin_queue=1868864, aggrutil=99.63%
  sdp: ios=545238/137415, merge=5086/405, ticks=1107748/761360, in_queue=1868864, util=99.63%

kvm

• ext4 rw,relatime,nobarrier,data=ordered,errors=remount-ro 0 1

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [28797KB/6663KB/0KB /s] [6617/1530/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=686: Tue Apr 12 10:12:28 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=19017KB/s, iops=3120, runt=176293msec
    slat (usec): min=1, max=93960, avg=53.38, stdev=181.43
    clat (usec): min=111, max=1536.1K, avg=16315.84, stdev=22454.79
     lat (usec): min=433, max=1536.1K, avg=16372.43, stdev=22449.84
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11200], 50.00th=[13888], 60.00th=[16512],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49408], 99.50th=[82432], 99.90th=[329728], 99.95th=[460800],
     | 99.99th=[749568]
    bw (KB  /s): min=  555, max=29968, per=100.00%, avg=19030.43, stdev=5338.76
  write: io=841688KB, bw=4774.4KB/s, iops=780, runt=176293msec
    slat (usec): min=2, max=29676, avg=67.29, stdev=300.20
    clat (usec): min=179, max=1534.6K, avg=16353.61, stdev=22471.60
     lat (usec): min=452, max=1534.6K, avg=16424.05, stdev=22467.04
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11328], 50.00th=[13888], 60.00th=[16768],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49920], 99.50th=[84480], 99.90th=[321536], 99.95th=[444416],
     | 99.99th=[741376]
    bw (KB  /s): min=   42, max= 7746, per=100.00%, avg=4778.38, stdev=1386.59
    lat (usec) : 250=0.01%, 500=0.05%, 750=0.32%, 1000=0.74%
    lat (msec) : 2=3.53%, 4=7.44%, 10=23.16%, 20=35.92%, 50=27.86%
    lat (msec) : 100=0.59%, 250=0.23%, 500=0.12%, 750=0.03%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=3.12%, sys=48.14%, ctx=319451, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=19017KB/s, minb=19017KB/s, maxb=19017KB/s, mint=176293msec, maxt=176293msec
  WRITE: io=841688KB, aggrb=4774KB/s, minb=4774KB/s, maxb=4774KB/s, mint=176293msec, maxt=176293msec

Disk stats (read/write):
  sda: ios=539868/137353, merge=10782/802, ticks=8688496/2847176, in_queue=11534452, util=100.00%

• default ext4 errors=remount-ro 0 1

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [22426KB/5431KB/0KB /s] [5174/1254/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=707: Tue Apr 12 10:04:43 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=27987KB/s, iops=4592, runt=119793msec
    slat (usec): min=2, max=8121, avg=138.38, stdev=78.30
    clat (usec): min=61, max=878540, avg=10981.79, stdev=26616.61
     lat (usec): min=205, max=878545, avg=11122.39, stdev=26610.60
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   19], 99.90th=[  627], 99.95th=[  693],
     | 99.99th=[  848]
    bw (KB  /s): min=  419, max=56905, per=100.00%, avg=29243.35, stdev=10428.05
  write: io=841688KB, bw=7026.2KB/s, iops=1149, runt=119793msec
    slat (usec): min=3, max=8183, avg=149.49, stdev=114.76
    clat (usec): min=564, max=878335, avg=11044.70, stdev=27526.77
     lat (usec): min=608, max=878344, avg=11196.21, stdev=27539.90
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   20], 99.90th=[  635], 99.95th=[  693],
     | 99.99th=[  873]
    bw (KB  /s): min=   69, max=14976, per=100.00%, avg=7341.69, stdev=2672.17
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.06%, 4=0.32%, 10=76.18%, 20=23.02%, 50=0.19%
    lat (msec) : 100=0.02%, 250=0.02%, 500=0.04%, 750=0.13%, 1000=0.02%
  cpu          : usr=2.64%, sys=81.63%, ctx=480152, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=27986KB/s, minb=27986KB/s, maxb=27986KB/s, mint=119793msec, maxt=119793msec
  WRITE: io=841688KB, aggrb=7026KB/s, minb=7026KB/s, maxb=7026KB/s, mint=119793msec, maxt=119793msec

Disk stats (read/write):
  sda: ios=548732/137547, merge=838/94, ticks=976940/269344, in_queue=1243892, util=95.91%

6 sata drive mirror with ssd log

I scrapped this set up. and went with a 2 drive mirror with the newest disks [ some of the others were 4-5 years old ].

kvm

jessie-template  ~ # fio /fbc/adm/fio.cfg  #  fio.cfg contains mir's code above . TBD upload fio.cfg or copy paste to screen
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [119.1MB/31420KB/0KB /s] [28.5K/7151/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1369: Mon Jun  6 12:49:29 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=164191KB/s, iops=26943, runt= 20419msec
    slat (usec): min=1, max=1030, avg= 6.10, stdev= 4.59
    clat (usec): min=117, max=166663, avg=1716.19, stdev=4230.12
     lat (usec): min=126, max=166670, avg=1722.55, stdev=4230.16
    clat percentiles (usec):
     |  1.00th=[  524],  5.00th=[  876], 10.00th=[  940], 20.00th=[  996],
     | 30.00th=[ 1032], 40.00th=[ 1064], 50.00th=[ 1096], 60.00th=[ 1144],
     | 70.00th=[ 1192], 80.00th=[ 1272], 90.00th=[ 1624], 95.00th=[ 2160],
     | 99.00th=[21120], 99.50th=[37632], 99.90th=[55040], 99.95th=[57600],
     | 99.99th=[62720]
    bw (KB  /s): min=95414, max=313106, per=100.00%, avg=164904.00, stdev=49364.41
  write: io=841688KB, bw=41221KB/s, iops=6740, runt= 20419msec
    slat (usec): min=2, max=44634, avg= 8.04, stdev=137.58
    clat (usec): min=419, max=89028, avg=2592.06, stdev=5228.52
     lat (usec): min=526, max=89034, avg=2600.37, stdev=5230.32
    clat percentiles (usec):
     |  1.00th=[ 1096],  5.00th=[ 1240], 10.00th=[ 1304], 20.00th=[ 1368],
     | 30.00th=[ 1416], 40.00th=[ 1464], 50.00th=[ 1512], 60.00th=[ 1576],
     | 70.00th=[ 1656], 80.00th=[ 1816], 90.00th=[ 2448], 95.00th=[ 4896],
     | 99.00th=[30848], 99.50th=[41216], 99.90th=[56576], 99.95th=[59136],
     | 99.99th=[61696]
    bw (KB  /s): min=23379, max=78212, per=100.00%, avg=41429.42, stdev=12385.69
    lat (usec) : 250=0.11%, 500=0.62%, 750=1.42%, 1000=14.84%
    lat (msec) : 2=74.72%, 4=5.20%, 10=0.72%, 20=1.05%, 50=1.12%
    lat (msec) : 100=0.21%, 250=0.01%
  cpu          : usr=11.13%, sys=25.59%, ctx=304291, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=164191KB/s, minb=164191KB/s, maxb=164191KB/s, mint=20419msec, maxt=20419msec
  WRITE: io=841688KB, aggrb=41220KB/s, minb=41220KB/s, maxb=41220KB/s, mint=20419msec, maxt=20419msec

Disk stats (read/write):
  vda: ios=548282/137250, merge=0/0, ticks=934124/351920, in_queue=1286136, util=99.50%