Hotplug (qemu disk,nic,cpu,memory): Difference between revisions

From Proxmox VE
Jump to navigation Jump to search
m (Update max_mem_regions when hotplug is for more than 64GB)
mNo edit summary
Line 5: Line 5:
- NICs (hotplug/unplug)
- NICs (hotplug/unplug)


- CPU (hotplug/unplug)
- USB devices (hotplug/unplug)
 
- CPU (hotplug/unplug)


- Memory (hotplug/unplug)
- Memory (hotplug/unplug)
Line 11: Line 13:
== Linux Guests Notes ==
== Linux Guests Notes ==


==== Kernel Modules Needed ====  
=== Kernel Modules Needed ===
You need to have 2 modules loaded in your Linux guest:
You need to have 2 modules loaded in your Linux guest:


<pre>
<pre>
#modprobe acpiphp
# modprobe acpiphp
#modprobe pci_hotplug
# modprobe pci_hotplug
</pre>
</pre>


==== CPU and Memory Hotplug ====  
=== CPU and Memory Hotplug ===  


You need a recent kernel (> 3.10) and a udev rules file, to auto-enable CPU or memory hotplugging.
==== Kernel older than 4.7 ====
You need a recent kernel (> 3.10) and an udev rules file, to auto-enable CPU or memory hot-plugging.


You can add the file:
You can add the file:


/lib/udev/rules.d/80-hotplug-cpu-mem.rules
<code>/lib/udev/rules.d/80-hotplug-cpu-mem.rules</code>
<pre>
<pre>
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"
Line 31: Line 34:
</pre>
</pre>


==== Kernel older than 5.15 ====
For Linux kernel >= 4.7, you don't need the udev rule for memory hotplug; you only need to add this kernel parameter at boot:
For Linux kernel >= 4.7, you don't need the udev rule for memory hotplug; you only need to add this kernel parameter at boot:


  memhp_default_state=online
  memhp_default_state=online


For Linux kernel >=5.15 (e.g. Ubuntu 22.04) you only need the UDEV rule for cpu hotplug:
==== Kernel 5.15 and newer ====
For Linux kernel >=5.15 (e.g. Ubuntu 22.04) you only need the UDEV rule for CPU hotplug:


/lib/udev/rules.d/80-hotplug-cpu.rules
/lib/udev/rules.d/80-hotplug-cpu.rules
Line 44: Line 49:
== Windows Guests Notes ==
== Windows Guests Notes ==


- Disk and NIC hotplug work out of the box for all Windows versions >2003
Disk and NIC hotplug work out of the box for all Windows versions >2003


- CPU and memory hotplug work on > 2008 enterprise/datacenter, 2012 > standard
CPU and memory hotplug work on > 2008 enterprise/datacenter, 2012 > standard


== Disk and NIC Hotplug/unplug ==
== Disk and NIC Hotplug/unplug ==


Edit your VM configuration file:
By default, hot-plug is already enabled for virtual network devices, virtual disks and virtual USB devices.
 
<pre>
/etc/pve/qemuserver/<VMID>.conf
</pre>
and add:
<pre>
hotplug: network,disk,....
</pre>


Note: Since Proxmox 3.3, the hotplug option is available in the GUI.
Just add/remove your disks, USB devices or NICs as usual.
 
Then add/remove your disks or NICs as usual.


For disk unplug, verify that your disk is not mounted in your Linux guest or enabled in Windows Disk Management.
For disk unplug, verify that your disk is not mounted in your Linux guest or enabled in Windows Disk Management.
Line 72: Line 67:
# You can now use the "CPU Options" dialog to change the number of vCPUs, while the machine is running.
# You can now use the "CPU Options" dialog to change the number of vCPUs, while the machine is running.


maxcpus = sockets * cores.
The maximum of CPUs equals sockets * cores.
 
Note : Windows CPU hotplug has been working since qemu 2.6 and CPU hot unplug has been working since qemu 2.7.


== Memory Hotplug ==
== Memory Hotplug ==


Memory hotplug is available in proxmox 3.4.
Memory hotplug is available since Proxmox VE 3.4.


You need to enable NUMA and hotplug to get it working:
To enable memory hotplug you need to enable both NUMA (web UI: VM -> Hardware -> CPU) and Memory hotplug (web UI: VM -> Options -> Hotplug).
 
<pre>
numa: 1
hotplug: memory,...
</pre>


Then you can use the memory GUI options as usual.
Then you can use the VMs memory options in the web UI as usual.


== Memory Hot Unplug ==
== Memory Hot Unplug ==


- Memory unplug currently does not work on Windows (<=win10 currently)
- Memory unplug currently does not work on Windows (<= win10 currently)


- Memory unplug can be unstable on Linux (<= kernel 4.10 currently)
- Memory unplug can be unstable on Linux (<= kernel 4.10 currently)
Line 103: Line 91:
Here are some future kernel patches to improve unplug : http://www.spinics.net/lists/linux-mm/msg125359.html
Here are some future kernel patches to improve unplug : http://www.spinics.net/lists/linux-mm/msg125359.html


== Update max_mem_regions when hotplug is for more than 64GB ==
== Update max_mem_regions when hot-plugging more than 64 GiB ==


<pre>
<pre>
For Error : a used vhost backend has no free memory slots left  
For Error: a used vhost backend has no free memory slots left  
</pre>
</pre>


Kernel sets how many memory dimms per vhost it supports, default 64.  
Kernel sets how many memory DIMMs per vhost it supports, default 64.  
You can increase the limit by creating a file /etc/modprobe.d/vhost.conf  
You can increase the limit by creating a file <code>/etc/modprobe.d/vhost.conf</code>.


Containing options vhost max_mem_regions=128 (default is 64, you can try increasing it further if necessary)  
Containing options vhost max_mem_regions=128 (default is 64, you can try increasing it further if necessary)  

Revision as of 07:15, 9 October 2024

Proxmox support hotplugging of Qemu devices:

- Disks (hotplug/unplug)

- NICs (hotplug/unplug)

- USB devices (hotplug/unplug)

- CPU (hotplug/unplug)

- Memory (hotplug/unplug)

Linux Guests Notes

Kernel Modules Needed

You need to have 2 modules loaded in your Linux guest:

# modprobe acpiphp
# modprobe pci_hotplug

CPU and Memory Hotplug

Kernel older than 4.7

You need a recent kernel (> 3.10) and an udev rules file, to auto-enable CPU or memory hot-plugging.

You can add the file:

/lib/udev/rules.d/80-hotplug-cpu-mem.rules

SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"
SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online"

Kernel older than 5.15

For Linux kernel >= 4.7, you don't need the udev rule for memory hotplug; you only need to add this kernel parameter at boot:

memhp_default_state=online

Kernel 5.15 and newer

For Linux kernel >=5.15 (e.g. Ubuntu 22.04) you only need the UDEV rule for CPU hotplug:

/lib/udev/rules.d/80-hotplug-cpu.rules

SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"

Windows Guests Notes

Disk and NIC hotplug work out of the box for all Windows versions >2003

CPU and memory hotplug work on > 2008 enterprise/datacenter, 2012 > standard

Disk and NIC Hotplug/unplug

By default, hot-plug is already enabled for virtual network devices, virtual disks and virtual USB devices.

Just add/remove your disks, USB devices or NICs as usual.

For disk unplug, verify that your disk is not mounted in your Linux guest or enabled in Windows Disk Management.

CPU Hotplug

  1. In the VM Options, edit the "Hotplug" Value and activate "CPU Hotplug".
  2. In the Hardware Section of your VM, specify the maximum number of Sockets/Cores.
  3. You can now use the "CPU Options" dialog to change the number of vCPUs, while the machine is running.

The maximum of CPUs equals sockets * cores.

Memory Hotplug

Memory hotplug is available since Proxmox VE 3.4.

To enable memory hotplug you need to enable both NUMA (web UI: VM -> Hardware -> CPU) and Memory hotplug (web UI: VM -> Options -> Hotplug).

Then you can use the VMs memory options in the web UI as usual.

Memory Hot Unplug

- Memory unplug currently does not work on Windows (<= win10 currently)

- Memory unplug can be unstable on Linux (<= kernel 4.10 currently)

For Linux memory unplug, you need to have movable zone enabled, in the kernel config (not enabled by default on Debian/Ubuntu):

CONFIG_MOVABLE_NODE=YES

and the "movable_node" boot kernel parameter enabled.

Here are some future kernel patches to improve unplug : http://www.spinics.net/lists/linux-mm/msg125359.html

Update max_mem_regions when hot-plugging more than 64 GiB

For Error: a used vhost backend has no free memory slots left 

Kernel sets how many memory DIMMs per vhost it supports, default 64. You can increase the limit by creating a file /etc/modprobe.d/vhost.conf.

Containing options vhost max_mem_regions=128 (default is 64, you can try increasing it further if necessary) and then rebooting the Proxmox machine.

echo "options vhost max_mem_regions=128" > /etc/modprobe.d/vhost.conf
rmmod vhost_net
rmmod vhost
modprobe vhost_net
cat /sys/module/vhost/parameters/max_mem_regions
# 128

Ref Forum link : https://forum.proxmox.com/threads/problem-with-hotplug-and-64gb-ram.73621/page-3#post-600595