[pve-devel] qemu memory hotplug limited to 64 dimm

[Wolfgang Bumiller]

So I see a few options one or more of which we should consider:

1) Make the *static* memory size configurable:

It doesn't make sense to me to eg. want to go from 1G to 200G in
small steps. Worse is the fact that even when starting with a higher
amount knowing that I'm only going to increase it, I start with lots and
lots of tiny dimms. Eg. 8G = 1G static + 14 x 512M dimms.

This could be added as a separate option or even to the existing
`memory` option without breaking existing configs (by turning it into a
property string with the current amount as the default key.)
I could then do:

  memory: 8192,static=8192

This example frees up 14 dimm slots alone.
The downside is this VM cannot go below 8G without a reboot.

The dimm size should then probably not start at 512M either. That's
where the next change comes in:

2) Change the steps one way or another:

  a) Generally change the number of steps before increasing the step
  size:
     The current 32 makes sense for the first few gigs, but if we eg. have
     a machine with 101G memory, we'd still be doing 2G steps. That
     amounts to a 2% granularity. At 113G we bump this up to 4G steps,
     which is a 3.5% granularity.
     I propose we use no more than 16 dimms of each size. (Heck I'd even
     be fine with 8 personally, although 14 of the 512M dimms would be
     good to get to a "nice" 8G in small steps. ;-) )

     This change is not backward compatible and would break memory
     hot-unplugging of running machines after the upgrade.
     Or well, it would still "happen", but our count would be out of
     sync.

  b) Make the _initial_ dimm size configurable:
     This would work well in combination with (1). (This initial dimm
     size would not be hot-pluggable as that would throw off our
     counter.)

  c) Have the option to use only 1 fixed dimm size.
     Since we already have a way to get a list of a running VM's dimms,
     this option could even be hot-pluggable. The unplug code would have
     to be changed to fetch the list of dimms from qemu rather than
     assume our fixed dimm list is used.
 [*] This would probably be a good idea in any case ;-)
     Of course this would allow the user to add a 16G dimm, change the
     dimm size to 1G and then try to reduce memory by 1 and fail.
     (I see no reason to protect people from too many of their own
     mistakes, but we *could* of course limit the hot-plugging of the
     dimm size to increasing only...)

  c) Use an explicit dimm list.
     This would be an alternative to (1), you define the static memory
     and the dimms, the latter can be hot plugged:

       memory: 8192,dimms=512x4:1024x4
     or
       memory: 8192
       dimms: 512x4 1024x4

     Would be the equivalent of 8G static memory plus 4 dimms of 512M
     and 1G each - a total of 14G.

     For backward compatibility we could require dimms to be "defined
     but empty" to use the new method, and keep the old method
     otherwise.

     Downside: who wants to make a GUI for that? ;-)

3) Drop support for hot*UN*plugging altogether and simply always hot-add
one big dimm. ;-)
On that note, my guests currently hang when I unplug memory and I bet
that happens to all of you and nobody ever noticed because nobody needs
that feature anyway ;-)