[pve-devel] qemu memory hotplug limited to 64 dimm

[Alexandre DERUMIER]

also see this redhat bugzilla, about kernel bug (even with online_movable)

https://bugzilla.redhat.com/show_bug.cgi?id=1320534

"
Description Milan Zamazal 2016-03-23 08:38:07 EDT
Description of problem:

File /usr/lib/udev/rules.d/40-redhat.rules contains the following line:

  SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"

This ensures that hotplugged memory gets available to the system.  In order to be able to hotunplug the hotplugged memory later, its state should be set to `online_movable' instead of `online'.  Otherwise the system may allocate kernel pages in the hotplugged memory, preventing its removal.  To avoid this problem the rule should look like

  SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online_movable"

However, there is a kernel bug that prevents this simple change from working, see https://bugzilla.redhat.com/1314306.  It's not possible to set memory blocks as online_movable in random order (it results in "Permission denied" failures), the state must be changed from the highest numbered plugged memory block to the lowest one.  So either the kernel bug should be fixed before the change, or the udev rules should ensure changing the state of the memory blocks to `online_movable' in the proper order (from the highest to the lowest memory block).

Version-Release number of selected component (if applicable):

219-19.el7_2.4.x86_64

"



----- Mail original -----
De: "Alexandre Derumier" <aderumier at odiso.com>
À: "Wolfgang Bumiller" <w.bumiller at proxmox.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Mercredi 14 Décembre 2016 17:05:20
Objet: Re: [pve-devel] qemu memory hotplug limited to 64 dimm

>>Apparently I do have to explicitly mention that I'm talking about *unused* memory, iow. memory where the /sys/../enabled file is 0. I think this was working at some point on modern guest >>kernels? 

do you mean "online" instead "enabled" ? 

also here some notes I have set in the wiki 



online should be set by an udev rules, to auto online dimm on hotplug 

/lib/udev/rules.d/80-hotplug-cpu-mem.rules 
SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1" 
SUBSYSTEM=="memory", ACTION=="add", TEST=="state", ATTR{state}=="offline", ATTR{state}="online" 

for linux kernel >= 4.7, 
you don't need the udev rule for memory hotplug, you only need to add this kernel parameter at boot: 

memhp_default_state=online 

Memory hot-unplug 
- Memory unplug don't work windows currently (<=win10 currently) 
- Memory unplug can't be unstable on linux (<= kernel 4.8 currently) 
for Linux memory unplug, you need have movable zone enabled in kernel config (not enabled by default on debian/ubuntu) 
CONFIG_MOVABLE_NODE=YES 
and 
"movable_node" boot kernel parameter enabled 



----- Mail original ----- 
De: "Wolfgang Bumiller" <w.bumiller at proxmox.com> 
À: "Alexandre Derumier" <aderumier at odiso.com> 
Cc: "pve-devel" <pve-devel at pve.proxmox.com>, "dietmar" <dietmar at proxmox.com> 
Envoyé: Mercredi 14 Décembre 2016 16:44:52 
Objet: Re: [pve-devel] qemu memory hotplug limited to 64 dimm 

> On December 14, 2016 at 3:45 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> 
> 
> >>3) Drop support for hot*UN*plugging altogether and simply always hot-add 
> >>one big dimm. ;-) 
> >>On that note, my guests currently hang when I unplug memory and I bet 
> >>that happens to all of you and nobody ever noticed because nobody needs 
> >>that feature anyway ;-) 
> 
> yes, it's a linux guest limitation. (qemu implementation is ok), but linux kernel memory is not movable currently, and if it's located on a dimm, you can't unplug it, as the kernel memory can't be move at another location. 

Apparently I do have to explicitly mention that I'm talking about *unused* memory, iow. memory where the /sys/../enabled file is 0. I think this was working at some point on modern guest kernels? 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel