[PVE-User] cores and threads Re: pve-user Digest, Vol 109, Issue 13

Tue Apr 18 22:31:43 CEST 2017

Former VMware employee here. Most of the concepts should carry over to KVM:

Contrary to what you might think, in most cases, having less cores per VM in an over-provisioned system will lead to an increase in performance. This is because in order for a VM with more than one core to process anything (even a single core task), the guest has to wait for a clock cycle to become available on all its cores at once. Many guests will panic or behave strangely if cores are allowed to process out of order. Furthermore, if one or more other guests with less cores are also queuing up tasks and can fit into a smaller number of cores, they may get priority and cause the higher-core guest to wait longer since they will be a more efficient use of CPU time from the host's perspective. This is referred to as co-stop, and is one of several silent killers of performance in a hypervisor.

Hyperthreading (or fake cores, as I lovingly refer to them), in my opinion, should never get passed to guests on a hypervisor, especially if the host is overprovisioned. The host can make better use of hyperthreading features than the guests can, and a guest that queues up work for more cores than there really are can lead to an even worse co-stop than what would normally be seen from just having too many multi-core VMs.

As for NUMA, you will be unlikely to see much performance differences on your hardware. The main purpose of NUMA is for multi-socket hosts that have guests/applications sharing memory between sockets. In that case, you will want to either set a guest to have cores that will fit within one socket (and its memory), or split the guest's cores/sockets/memory evenly between as few physical sockets as possible. The more physical sockets that are involved, the more processing latency is added due to memory copying over a slower bus when a process needs to access active memory from a neighboring NUMA node. For single-socket systems like yours, NUMA might enable a few features like dynamically adding RAM and cores to the guest, but not really have much impact on performance.

My advice would be to add more VMs with as few cores as possible and add a load-balancer like HAproxy where applicable. 

Again, that's the knowledge I gleaned for ESXi while doing performance tuning in large systems. If anyone has contrary information with respect to how KVM does things, I would be very interested to learn more.

 - Jeremy

----- Original Message -----
From: "Michael Peele" <peele1 at gmail.com>
To: pve-user at pve.proxmox.com
Sent: Tuesday, April 18, 2017 11:07:19 AM
Subject: [PVE-User] cores and threads Re: pve-user Digest, Vol 109, Issue 13

The common thinking is do not assign more than N-1 cores to a VM (to allow
the VM to have a core).
If you have multiple VMs, then it will really vary a lot on what you can
assign.  You might try giving the high CPU one 4 cores, the moderate 3, and
the low ones 2 each.  Yes, that is 9.  You can oversubscribe, just don't
give one VM too many (N-1; in your case 7, though only 4 are real, which
makes it more complex.)

It is very complex with real cores, hyperthread cores, CPU load, memory
load, etc.

If a VM is maxing out, give it another core, if you want.  If a VM is very
low on CPU, maybe remove a core, but if it never uses much, it won't
matter.  Assigning cores is more about CPU prioritization and VM
prioritization.

Maybe you determine that DNS, while  normally having low usage, needs to
take some random DoS, so you give DNS 7 cores even though it normally only
needs 1.  Maybe the VM that has 100% CPU usage on 2 cores is a "low"
priority test system, so you won't give it more than 2 just because.

Thus, nobody can tell you what your combination of supply (ram, CPU, disk,
network), demand, and priorities are.

Regards.

On Sun, Apr 16, 2017 at 6:00 AM, <pve-user-request at pve.proxmox.com> wrote:

> Send pve-user mailing list submissions to
>         pve-user at pve.proxmox.com
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> or, via email, send a message with subject or body 'help' to
>         pve-user-request at pve.proxmox.com
>
> You can reach the person managing the list at
>         pve-user-owner at pve.proxmox.com
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of pve-user digest..."
>
>
> Today's Topics:
>
>    1. Re: cores and threads (Miguel González)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 15 Apr 2017 22:01:32 +0200
> From: Miguel González <miguel_3_gonzalez at yahoo.es>
> To: "pve-user at pve.proxmox.com" <pve-user at pve.proxmox.com>
> Subject: Re: [PVE-User] cores and threads
> Message-ID: <c0336d43-8f5c-abb8-041c-652a7c0953bd at yahoo.es>
> Content-Type: text/plain; charset=utf-8
>
> No one can help me out?
>
>
> On 04/13/17 4:55 PM, Miguel González wrote:
> > Hi,
> >
> >   I have a Proxmox 4 host running Intel Xeon W3520 with 4 cores and 8
> > threads with 32 gb of RAM. This host has 5 VMs.
> >
> >   Two VMs has quite heavy CPU usage (two web servers), another one quite
> > moderate (email server, but only with a few accounts) and two DNS
> > servers with low usage.
> >
> >   How many sockets and cores can I allocate?
> >
> >   Can I enable NUMA? I get:
> >
> >   numactl --hardware
> > available: 1 nodes (0)
> > node 0 cpus: 0 1 2 3 4 5 6 7
> > node 0 size: 32165 MB
> > node 0 free: 164 MB
> > node distances:
> > node   0
> >   0:  10
> >
> >
> >   I have searched on Google and the information is not very clear about
> > how many sockets and cores I can set in a VM.
> >
> >   Thanks!
> >
> >   Miguel
> >
> >
> >
> >
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
> ------------------------------
>
> End of pve-user Digest, Vol 109, Issue 13
> *****************************************
>
_______________________________________________
pve-user mailing list
pve-user at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

-- 
This message has been scanned for viruses and dangerous content by 
E.F.A. Project, and is believed to be clean.

Click here to report this message as spam.
http://spam.awesomegeek.com/cgi-bin/learn-msg.cgi?id=43A18100722.ABF2C&token=3412b99810beb45f040133d427f211aa