Iscsi/tests

From Proxmox VE
Revision as of 11:53, 29 November 2016 by Wolfgang Link (talk | contribs) (→‎= 2 drive sata mirror)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

fio

for more info on fio see: TBD 

apt-get install fio

create /root/fio.cfg : 

echo "# from mir pve forum
[global]
description=Emulation of Intel IOmeter File Server Access Pattern

[iometer]
bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10
rw=randrw
rwmixread=80
direct=1
size=4g
ioengine=libaio
# IOMeter defines the server loads as the following:
# iodepth=1    Linear
# iodepth=4    Very Light
# iodepth=8    Light
# iodepth=64    Moderate
# iodepth=256    Heavy
iodepth=64" > /root/fio.cfg

mir's tests

  • without nobarrier

/dev/sdb1 on /media/disk type ext4 (rw,relatime,data=ordered) 

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=18596KB/s, minb=18,596KB/s, maxb=18,596KB/s, mint=18,0281msec, maxt=18,0281msec
WRITE: io=841688KB, aggrb=4668KB/s, minb=4,668KB/s, maxb=4,668KB/s, mint=18,0281msec, maxt=180281msec

Disk stats (read/write):
sdb: ios=549723/137605, merge=1/36, ticks=9265908/2204204, in_queue=11639448, util=100.00%
  • nobarrier

/dev/sdb1 on /media/disk type ext4 (rw,relatime,nobarrier,data=ordered) 

Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=30,526KB/s, minb=30,526KB/s, maxb=30,526KB/s, mint=10,9826msec, maxt=10,9826msec
WRITE: io=841688KB, aggrb=7,663KB/s, minb=7,663KB/s, maxb=7,663KB/s, mint=10,9826msec, maxt=10,9826msec

Disk stats (read/write):
sdb: ios=550055/137665, merge=13/24, ticks=5510488/1558668, in_queue=7071272, util=100.00%

using XFS as filesystem inside the VM.

/dev/sdb1 on /media/disk type xfs (rw,relatime,attr2,inode64,noquota) 


Run status group 0 (all jobs):
READ: io=3274.5MB, aggrb=38,831KB/s, minb=38,831KB/s, maxb=38,831KB/s, mint=86,337msec, maxt=86,337msec
WRITE: io=841688KB, aggrb=9,748KB/s, minb=9,748KB/s, maxb=9,748KB/s, mint=86337msec, maxt=86337msec

Disk stats (read/write):
sdb: ios=549888/137583, merge=0/3, ticks=375584/77740, in_queue=452892, util=94.83%

sys4 tests

napp-it/omnios , motherboard: supermicro x9xcl-f , memory 28GB , cpu: Xeon(R) CPU E3-1280 V2 @ 3.60GHz [htop shows 8 cores ]

two SAS2008 IT mode HBA cards

this testing system had multipath issues. will try to redo tests on final system.

the following tests use a debian jessie virtual machines.


raidz1 with a log drive

zfs: raidz1 : 5 intel ssd pro series 2500 480GB + zil intel ssd s3700

lxc

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [252.7MB/65052KB/0KB /s] [59.3K/14.8K/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=2513: Sat Apr  9 15:24:14 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=303130KB/s, iops=49742, runt= 11060msec
  slat (usec): min=2, max=588, avg= 7.45, stdev=18.05
  clat (usec): min=160, max=252974, avg=885.66, stdev=2765.50
  lat (usec): min=167, max=252977, avg=893.38, stdev=2765.43
  clat percentiles (usec):
  |  1.00th=[  334],  5.00th=[  406], 10.00th=[  482], 20.00th=[  580],
  | 30.00th=[  652], 40.00th=[  716], 50.00th=[  772], 60.00th=[  836],
  | 70.00th=[  892], 80.00th=[  980], 90.00th=[ 1128], 95.00th=[ 1304],
  | 99.00th=[ 1832], 99.50th=[ 2320], 99.90th=[20608], 99.95th=[38656],
  | 99.99th=[136192]
  bw (KB  /s): min=118739, max=444443, per=100.00%, avg=305665.86, stdev=75220.28
  write: io=841688KB, bw=76102KB/s, iops=12445, runt= 11060msec
  slat (usec): min=3, max=1020, avg= 9.17, stdev=20.35
  clat (usec): min=395, max=264907, avg=1552.83, stdev=5218.00
  lat (usec): min=402, max=264915, avg=1562.29, stdev=5217.88
  clat percentiles (usec):
  |  1.00th=[  644],  5.00th=[  804], 10.00th=[  908], 20.00th=[ 1032],
  | 30.00th=[ 1128], 40.00th=[ 1208], 50.00th=[ 1288], 60.00th=[ 1384],
  | 70.00th=[ 1480], 80.00th=[ 1624], 90.00th=[ 1864], 95.00th=[ 2128],
  | 99.00th=[ 2960], 99.50th=[ 4320], 99.90th=[61696], 99.95th=[121344],
  | 99.99th=[252928]
  bw (KB  /s): min=28575, max=111024, per=100.00%, avg=76804.24, stdev=18922.29
  lat (usec) : 250=0.01%, 500=9.42%, 750=27.66%, 1000=31.51%
  lat (msec) : 2=29.46%, 4=1.58%, 10=0.12%, 20=0.06%, 50=0.12%
  lat (msec) : 100=0.02%, 250=0.03%, 500=0.01%
  cpu  : usr=12.19%, sys=56.86%, ctx=17722, majf=0, minf=8
  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
  submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
  issued  : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
  latency  : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: io=3274.5MB, aggrb=303130KB/s, minb=303130KB/s, maxb=303130KB/s, mint=11060msec, maxt=11060msec
  WRITE: io=841688KB, aggrb=76101KB/s, minb=76101KB/s, maxb=76101KB/s, mint=11060msec, maxt=11060msec

Disk stats (read/write):
  dm-2: ios=543761/136160, merge=0/0, ticks=349496/178408, in_queue=527996, util=99.14%, aggrios=547260/137478, aggrmerge=2955/219, aggrticks=351272/179892, aggrin_queue=530992, aggrutil=98.92%
  sdk: ios=547260/137478, merge=2955/219, ticks=351272/179892, in_queue=530992, util=98.92%

kvm

/dev/sda1 on / type ext4 (rw,relatime,nobarrier,data=ordered)

at pve cache set to default nocache. 

iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [13251KB/3173KB/0KB /s] [3093/735/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=695: Sat Apr  9 19:12:16 2016   
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]   
  read : io=3274.5MB, bw=13694KB/s, iops=2247, runt=244825msec   
  slat (usec): min=1, max=34337, avg=10.52, stdev=160.01
  clat (usec): min=96, max=485692, avg=22701.29, stdev=14578.86
  lat (usec): min=660, max=485701, avg=22712.10, stdev=14578.48
  clat percentiles (usec):
  |  1.00th=[ 1112],  5.00th=[ 2800], 10.00th=[ 4960], 20.00th=[ 9152],
  | 30.00th=[13504], 40.00th=[17792], 50.00th=[22144], 60.00th=[26496],
  | 70.00th=[30848], 80.00th=[35072], 90.00th=[40192], 95.00th=[43264],
  | 99.00th=[52480], 99.50th=[60160], 99.90th=[85504], 99.95th=[113152],
  | 99.99th=[296960]
  bw (KB  /s): min= 5004, max=25066, per=100.00%, avg=13713.01, stdev=3454.14
  write: io=841688KB, bw=3437.1KB/s, iops=562, runt=244825msec
  slat (usec): min=3, max=32123, avg=17.92, stdev=409.82
  clat (usec): min=868, max=483835, avg=23016.88, stdev=14330.54
  lat (usec): min=881, max=483848, avg=23035.12, stdev=14335.88
  clat percentiles (usec):
  |  1.00th=[ 1416],  5.00th=[ 3056], 10.00th=[ 5216], 20.00th=[ 9536],
  | 30.00th=[13888], 40.00th=[18304], 50.00th=[22400], 60.00th=[26752],
  | 70.00th=[31104], 80.00th=[35584], 90.00th=[40192], 95.00th=[43776],
  | 99.00th=[53504], 99.50th=[61184], 99.90th=[85504], 99.95th=[98816],
  | 99.99th=[264192]
  bw (KB  /s): min= 1422, max= 6579, per=100.00%, avg=3443.28, stdev=926.27
  lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.51%
  lat (msec) : 2=2.47%, 4=4.68%, 10=13.98%, 20=23.24%, 50=53.71%
  lat (msec) : 100=1.32%, 250=0.04%, 500=0.02%
  cpu  : usr=10.73%, sys=23.07%, ctx=673631, majf=0, minf=8
  IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
  submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
  issued  : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
  latency  : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  READ: io=3274.5MB, aggrb=13693KB/s, minb=13693KB/s, maxb=13693KB/s, mint=244825msec, maxt=244825msec
  WRITE: io=841688KB, aggrb=3437KB/s, minb=3437KB/s, maxb=3437KB/s, mint=244825msec, maxt=244825msec

Disk stats (read/write):
  sda: ios=539489/137206, merge=11367/712, ticks=12209328/3240400, in_queue=15451416, util=100.00%

4 drive mirror, without a log drive

four intel ssd pro series 2500 480GB attached to its own HBA.

lxc

3032-jessie  ~ # fio /root/fio.cfg
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [93189KB/23350KB/0KB /s] [21.7K/5469/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1857: Tue Apr 12 09:39:34 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=109922KB/s, iops=18037, runt= 30500msec
    slat (usec): min=2, max=8635, avg= 6.00, stdev=16.12
    clat (usec): min=144, max=43100, avg=2123.05, stdev=2388.69
     lat (usec): min=166, max=43103, avg=2129.27, stdev=2388.60
    clat percentiles (usec):
     |  1.00th=[  318],  5.00th=[  418], 10.00th=[  486], 20.00th=[  604],
     | 30.00th=[  732], 40.00th=[  892], 50.00th=[ 1160], 60.00th=[ 1624],
     | 70.00th=[ 2384], 80.00th=[ 3312], 90.00th=[ 4896], 95.00th=[ 6496],
     | 99.00th=[10816], 99.50th=[13248], 99.90th=[22144], 99.95th=[24192],
     | 99.99th=[31616]
    bw (KB  /s): min=79049, max=218569, per=100.00%, avg=110088.87, stdev=30276.89
  write: io=841688KB, bw=27596KB/s, iops=4512, runt= 30500msec
    slat (usec): min=3, max=8921, avg= 7.78, stdev=35.57
    clat (usec): min=462, max=54371, avg=5654.28, stdev=3658.82
     lat (usec): min=469, max=54379, avg=5662.30, stdev=3658.89
    clat percentiles (usec):
     |  1.00th=[ 1032],  5.00th=[ 1448], 10.00th=[ 1880], 20.00th=[ 2704],
     | 30.00th=[ 3408], 40.00th=[ 4128], 50.00th=[ 4896], 60.00th=[ 5728],
     | 70.00th=[ 6752], 80.00th=[ 8096], 90.00th=[10304], 95.00th=[12352],
     | 99.00th=[17536], 99.50th=[21632], 99.90th=[28288], 99.95th=[31104],
     | 99.99th=[41216]
    bw (KB  /s): min=19175, max=53703, per=100.00%, avg=27661.45, stdev=7629.12
    lat (usec) : 250=0.09%, 500=8.83%, 750=16.25%, 1000=10.74%
    lat (msec) : 2=18.74%, 4=21.13%, 10=20.97%, 20=2.96%, 50=0.31%
    lat (msec) : 100=0.01%
  cpu          : usr=5.42%, sys=19.14%, ctx=91520, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=109921KB/s, minb=109921KB/s, maxb=109921KB/s, mint=30500msec, maxt=30500msec
  WRITE: io=841688KB, aggrb=27596KB/s, minb=27596KB/s, maxb=27596KB/s, mint=30500msec, maxt=30500msec

Disk stats (read/write):
    dm-3: ios=547778/137138, merge=0/0, ticks=1112764/760488, in_queue=1873360, util=99.74%, aggrios=545238/137415, aggrmerge=5086/405, aggrticks=1107748/761360, aggrin_queue=1868864, aggrutil=99.63%
  sdp: ios=545238/137415, merge=5086/405, ticks=1107748/761360, in_queue=1868864, util=99.63%

kvm

  • ext4 rw,relatime,nobarrier,data=ordered,errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [28797KB/6663KB/0KB /s] [6617/1530/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=686: Tue Apr 12 10:12:28 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=19017KB/s, iops=3120, runt=176293msec
    slat (usec): min=1, max=93960, avg=53.38, stdev=181.43
    clat (usec): min=111, max=1536.1K, avg=16315.84, stdev=22454.79
     lat (usec): min=433, max=1536.1K, avg=16372.43, stdev=22449.84
    clat percentiles (usec):
     |  1.00th=[  964],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11200], 50.00th=[13888], 60.00th=[16512],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49408], 99.50th=[82432], 99.90th=[329728], 99.95th=[460800],
     | 99.99th=[749568]
    bw (KB  /s): min=  555, max=29968, per=100.00%, avg=19030.43, stdev=5338.76
  write: io=841688KB, bw=4774.4KB/s, iops=780, runt=176293msec
    slat (usec): min=2, max=29676, avg=67.29, stdev=300.20
    clat (usec): min=179, max=1534.6K, avg=16353.61, stdev=22471.60
     lat (usec): min=452, max=1534.6K, avg=16424.05, stdev=22467.04
    clat percentiles (usec):
     |  1.00th=[  956],  5.00th=[ 2096], 10.00th=[ 3440], 20.00th=[ 6048],
     | 30.00th=[ 8640], 40.00th=[11328], 50.00th=[13888], 60.00th=[16768],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[29312], 95.00th=[34560],
     | 99.00th=[49920], 99.50th=[84480], 99.90th=[321536], 99.95th=[444416],
     | 99.99th=[741376]
    bw (KB  /s): min=   42, max= 7746, per=100.00%, avg=4778.38, stdev=1386.59
    lat (usec) : 250=0.01%, 500=0.05%, 750=0.32%, 1000=0.74%
    lat (msec) : 2=3.53%, 4=7.44%, 10=23.16%, 20=35.92%, 50=27.86%
    lat (msec) : 100=0.59%, 250=0.23%, 500=0.12%, 750=0.03%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=3.12%, sys=48.14%, ctx=319451, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=19017KB/s, minb=19017KB/s, maxb=19017KB/s, mint=176293msec, maxt=176293msec
  WRITE: io=841688KB, aggrb=4774KB/s, minb=4774KB/s, maxb=4774KB/s, mint=176293msec, maxt=176293msec

Disk stats (read/write):
  sda: ios=539868/137353, merge=10782/802, ticks=8688496/2847176, in_queue=11534452, util=100.00%
  • default ext4 errors=remount-ro 0 1
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [22426KB/5431KB/0KB /s] [5174/1254/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=707: Tue Apr 12 10:04:43 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=27987KB/s, iops=4592, runt=119793msec
    slat (usec): min=2, max=8121, avg=138.38, stdev=78.30
    clat (usec): min=61, max=878540, avg=10981.79, stdev=26616.61
     lat (usec): min=205, max=878545, avg=11122.39, stdev=26610.60
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   19], 99.90th=[  627], 99.95th=[  693],
     | 99.99th=[  848]
    bw (KB  /s): min=  419, max=56905, per=100.00%, avg=29243.35, stdev=10428.05
  write: io=841688KB, bw=7026.2KB/s, iops=1149, runt=119793msec
    slat (usec): min=3, max=8183, avg=149.49, stdev=114.76
    clat (usec): min=564, max=878335, avg=11044.70, stdev=27526.77
     lat (usec): min=608, max=878344, avg=11196.21, stdev=27539.90
    clat percentiles (msec):
     |  1.00th=[    6],  5.00th=[    8], 10.00th=[    9], 20.00th=[   10],
     | 30.00th=[   10], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
     | 99.00th=[   17], 99.50th=[   20], 99.90th=[  635], 99.95th=[  693],
     | 99.99th=[  873]
    bw (KB  /s): min=   69, max=14976, per=100.00%, avg=7341.69, stdev=2672.17
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
    lat (msec) : 2=0.06%, 4=0.32%, 10=76.18%, 20=23.02%, 50=0.19%
    lat (msec) : 100=0.02%, 250=0.02%, 500=0.04%, 750=0.13%, 1000=0.02%
  cpu          : usr=2.64%, sys=81.63%, ctx=480152, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=27986KB/s, minb=27986KB/s, maxb=27986KB/s, mint=119793msec, maxt=119793msec
  WRITE: io=841688KB, aggrb=7026KB/s, minb=7026KB/s, maxb=7026KB/s, mint=119793msec, maxt=119793msec

Disk stats (read/write):
  sda: ios=548732/137547, merge=838/94, ticks=976940/269344, in_queue=1243892, util=95.91%

6 sata drive mirror with ssd log

I scrapped this set up. and went with a 2 drive mirror with the newest disks [ some of the others were 4-5 years old ].

kvm 

jessie-template  ~ # fio /fbc/adm/fio.cfg  #  fio.cfg contains mir's code above . TBD upload fio.cfg or copy paste to screen
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
iometer: Laying out IO file(s) (1 file(s) / 4096MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [119.1MB/31420KB/0KB /s] [28.5K/7151/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=1369: Mon Jun  6 12:49:29 2016
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3274.5MB, bw=164191KB/s, iops=26943, runt= 20419msec
    slat (usec): min=1, max=1030, avg= 6.10, stdev= 4.59
    clat (usec): min=117, max=166663, avg=1716.19, stdev=4230.12
     lat (usec): min=126, max=166670, avg=1722.55, stdev=4230.16
    clat percentiles (usec):
     |  1.00th=[  524],  5.00th=[  876], 10.00th=[  940], 20.00th=[  996],
     | 30.00th=[ 1032], 40.00th=[ 1064], 50.00th=[ 1096], 60.00th=[ 1144],
     | 70.00th=[ 1192], 80.00th=[ 1272], 90.00th=[ 1624], 95.00th=[ 2160],
     | 99.00th=[21120], 99.50th=[37632], 99.90th=[55040], 99.95th=[57600],
     | 99.99th=[62720]
    bw (KB  /s): min=95414, max=313106, per=100.00%, avg=164904.00, stdev=49364.41
  write: io=841688KB, bw=41221KB/s, iops=6740, runt= 20419msec
    slat (usec): min=2, max=44634, avg= 8.04, stdev=137.58
    clat (usec): min=419, max=89028, avg=2592.06, stdev=5228.52
     lat (usec): min=526, max=89034, avg=2600.37, stdev=5230.32
    clat percentiles (usec):
     |  1.00th=[ 1096],  5.00th=[ 1240], 10.00th=[ 1304], 20.00th=[ 1368],
     | 30.00th=[ 1416], 40.00th=[ 1464], 50.00th=[ 1512], 60.00th=[ 1576],
     | 70.00th=[ 1656], 80.00th=[ 1816], 90.00th=[ 2448], 95.00th=[ 4896],
     | 99.00th=[30848], 99.50th=[41216], 99.90th=[56576], 99.95th=[59136],
     | 99.99th=[61696]
    bw (KB  /s): min=23379, max=78212, per=100.00%, avg=41429.42, stdev=12385.69
    lat (usec) : 250=0.11%, 500=0.62%, 750=1.42%, 1000=14.84%
    lat (msec) : 2=74.72%, 4=5.20%, 10=0.72%, 20=1.05%, 50=1.12%
    lat (msec) : 100=0.21%, 250=0.01%
  cpu          : usr=11.13%, sys=25.59%, ctx=304291, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=550156/w=137644/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3274.5MB, aggrb=164191KB/s, minb=164191KB/s, maxb=164191KB/s, mint=20419msec, maxt=20419msec
  WRITE: io=841688KB, aggrb=41220KB/s, minb=41220KB/s, maxb=41220KB/s, mint=20419msec, maxt=20419msec

Disk stats (read/write):
  vda: ios=548282/137250, merge=0/0, ticks=934124/351920, in_queue=1286136, util=99.50%

2 drive sata mirror

with zil and cache on a intel super ssd [model later if needed, 200GB recommended model ]

Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Iscsi/tests&oldid=9538"