[PVE-User] RAID array performance measuring

[Frederic Van Espen]

Hi,

We currently have a 2-node cluster running on 2 almost identical machines:
- 2 quad-core AMD opteron 2350
- 16 GB / 32 GB of ram
- RAID5 with 8 300GB SAS 15k disks / RAID5 with 4 500GB SAS 15k disks
- RAID5 is running on ADAPTEC 5805 RAID controllers, backed with a battery.

The more powerful node is currently running 42 KVM machines. The other 
one is running 29 KVM machines (we don't use OpenVZ).

Lately, the performance of these machines is feeling rather bad, so we 
have decided to add another node:
HP DL360 G8
- 2 quad-core Intel XEON E5-2609 cpu's
- 64 GB ram
- RAID5 with 5 300 GB SAS 15k disks
- RAID5 is running on an HP smart array P420i controller, backed with a 
battery

Currently only one KVM machine is running on this as we are still 
test-driving the hardware

The write cache on all of these is set to write-back (as reported by the 
raid management tools)

These nodes do not use shared storage, but each of them has its own LVM 
group on the RAID5 dedicated.

However, when I run pveperf to measure performance, on the old nodes I 
get output similar to this:

vm004:~# pveperf
CPU BOGOMIPS:      32161.80
REGEX/SECOND:      777714
HD SIZE:           3.66 GB (/dev/mapper/vm004-ROOT)
BUFFERED READS:    236.21 MB/sec
AVERAGE SEEK TIME: 4.18 ms
FSYNCS/SECOND:     199.03

vm005:~# pveperf
CPU BOGOMIPS:      35381.72
REGEX/SECOND:      835023
HD SIZE:           3.66 GB (/dev/mapper/vm005-ROOT)
BUFFERED READS:    145.70 MB/sec
AVERAGE SEEK TIME: 13.15 ms
FSYNCS/SECOND:     174.94

Note the low FSYNCS/SECOND output.

Now on the new node:

root at vm007:~# pveperf
CPU BOGOMIPS:      38307.16
REGEX/SECOND:      1113378
HD SIZE:           2.75 GB (/dev/mapper/vm007-vm007--ROOT)
BUFFERED READS:    715.10 MB/sec
AVERAGE SEEK TIME: 3.90 ms
FSYNCS/SECOND:     3561.76

FSYNC performance on this node is roughly 18 times better. A few 
questions to anyone that can provide some insight:
- Can this be caused simply by the higher load on the older nodes? E.g, 
if I try to turn off all KVM machines, will the output of pveperf be any 
different?
- If no to the above question, How should I proceed to pin down what is 
causing this issue?

Thanks,

Frederic