High Availability Cluster 4.x
Introduction
Proxmox VE High Availability Cluster (Proxmox VE HA Cluster) enables the definition of high available virtual machines. In simple words, if a virtual machine (VM) is configured as HA and the physical host fails, the VM is automatically restarted on one of the remaining Proxmox VE Cluster nodes.
The Proxmox VE HA Cluster is based on the Proxmox VE HA Manager (pve-ha-manager) - using watchdog fencing. Major benefit of Linux softdog or hardware watchdog is zero configuration - it just works out of the box.
In order to learn more about functionality of the new Proxmox VE HA manager, install the HA simulator.
Update to the latest version
Before you start, make sure you have installed the latest packages.
Note: if you have already a running HA setup update the Node which is the current master at last, this makes the update process a lot faster. Then run:
apt-get update && apt-get dist-upgrade
System requirements
If you run HA, high end server hardware with no single point of failure is required. This includes redundant disks, redundant power supply, UPS systems, and network bonding.
- Proxmox VE 4.0 comes with Self-Fencing with hardware watchdog or Software watchdog.
- Fully configured Proxmox_VE_4.x_Cluster (version 4.0 and later), with at least 3 nodes (maximum supported configuration: currently 32 nodes per cluster).
- Shared storage (SAN/iSCSI, NAS/NFS, Ceph, DRBD9... for virtual disk images)
- Reliable, redundant network, suitable configured which supports multicast
- An extra network for cluster communication, one network for VM traffic and one network for storage traffic.
It's essential that you use redundant network connections for the cluster communication (bonding). If not, a simple switch reboot (or power loss on the switch) can fence all cluster nodes if it takes longer than 60 sec.
HA Configuration
Adding and managing VM´s or containers for HA can be done via GUI or CLI (`ha-manager add <VMID>`).
Important: note that before enabling HA for a service you should test it thoughtfully. See if migration works, look that NO local resources are used by it. Secure that it may run on all nodes defined by its group and even better on all cluster nodes.
Fencing
Proxmox VE Cluster 4.0 or greater comes with watchdog fencing. This works out of the box, no configuration is required.
How Watchdog fencing works:
If the node has connection with the cluster and has quorum, the watchdog will be reset. If quorum is lost, the node is not able to reset the watchdog. This will trigger a reboot after 60 seconds.
If your hardware has a hardware watchdog, this one will be automatically detected and used. Otherwise, ha-manager just uses the Linux softdog. Therefore testing Proxmox VE HA inside a virtual environment is possible.
Permissions
From version 1.0-13 of the pve-ha-manager package, the HA stack is better integrated in the permission system of Proxmox VE.
- Creation, deletion and updating a resource or group needs the 'Sys.Console' privilege for the whole cluster (i.e. on the root path '/').
- The current status and the configured resources and groups may be read (but not written) with the 'Sys.Audit' privilege on the root path.
HA Groups
The Proxmox VE HA Cluster is using groups for mapping VMs to a set of node.
For example: If a "vm100" is in the group "ONE" and group "ONE" has members "pve1,pve2" and "vm100" is running on pve1.
When "pve1" is crashing. "vm100" will migrated to "pve2".
The Proxmox VE HA Groups has two option restricted and nofailback.
- restricted: VM's bound to the group may only run on cluster members which are also members of the group. If no members of the group are available, the service is placed in the stopped state.
- nofailback: Enabling this option for a group will prevent automated fail-back after a more-preferred node rejoins the cluster.
Further you may set relative priorities for each node in a group, the VMs run then on the node(s) with the highest priority.
As an example imagine we have a 5 node cluster, two big nodes big-1, big-2, and three lighter nodes light-1, light-2 and light-3. We want that our important VMs assigned to the group "production" run on the big nodes if possible, only if both big nodes fail we chose our lighter ones for recovery. That scenario can be configure by setting the "production" groups node list like:
ha-manager groupset production -nodes 'big-1:2,big-2:2,light-1:1,light-2:1,light-3:1'
So you have a comma separated list for the node names and further you may add an priority as an integer after each node name, separated with a colon.
If you do not set explicitly a priority the ha-manager assumes implicit that its 0.
Priority configuration cannot be done via GUI yet, but is in active work.
Enable a VM/CT for HA
On the CLI, you can use ha-manger to achieve this task.
IMPORTANT:
If you enable HA it's not possible to turnoff the VM inside the VM. Also, if it is disabled the VM will be stopped.
If you add a VM/CT, its instantly 'ha-managed'.
ha-manager add vm:100
To add a VM/CT on GUI.
Disable a VM/CT for HA
If you want to disable a ha-managed VM/CT (e.g. for shutdown) via CLI:
ha-manager disable vm:100
If you want to re-enable a ha-managed VM/CT:
ha-manager enable vm:100
HA Cluster Maintenance (node reboots)
If you need to reboot a node, e.g. because of a kernel update, you need to migrate all VM/CT to another node or disable them. By disabling them, all resources are stopped. All VM guests will get an ACPI shutdown request, if this won't work due to VM internal settings, they'll be forced to stop after a minute.
As soon as all VM/CT are either stopped or migrated reboot your node. Then when the node is up again, continue with doing the same on the next node and so on.
Graceful shutdown/reboot behaviour
Note (pve-ha-manger version < 1.0-15): The services won't get automatically migrated by the HA stack. You have to migrate them manually before you power off your node (for example for hardware maintenance). The LRM will instead freeze the services which are still on the node in the expectation that the Node will come soon up again.
Note (pve-ha-manger version >= 1.0-15): The Services will get stopped (but not disabled) but not touched in any other way. That means that if the Node had enabled Services they will get fenced and relocated if the powered down stays off for about more then two minutes.
HA Simulator
By using the HA simulator you can test and learn all functionalities of the Proxmox VE HA solutions.
The simulator allows you to watch and test the behaviour of a real-world 3 node cluster with 6 VM's.
You do not have to setup or configure a real cluster, the HA simulator runs out of the box on the current code base.
Install with apt:
apt-get install pve-ha-simulator
To start the simulator you must have a X11 redirection to your current system.
If you are on a Linux machine you can use:
ssh root@<IPofPVE4> -Y
On Windows it is working with mobaxterm.
After starting the simulator create a working directory:
mkdir working
To start the simulator type
pve-ha-simulator working/
Hardware Watchdogs
if no hardware watchdog is defined, proxmox is loading the softdog module, which emulate the /dev/watchdog device.
To enable a hardware watchdog, you need to specify the module to load
/etc/default/pve-ha-manager WATCHDOG_MODULE=mywatchdogmodule
also, please disable nmi watchdog, which is embedeed in cpu apic.
edit: /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet nmi_watchdog=0"
the
# update-grub
iTCO Watchdog (module "iTCO_wdt")
This is an hardware watchdog, available in almost all intels motherboard (ich chipset) since 15 years.
IPMI Watchdog (module "ipmi_watchdog")
For IPMI Watchdogs you may have to set the action, else it may not do anything when it triggers.
For this purpose edit the /etc/modprobe.d/ipmi_watchdog.conf (simple create the file):
options ipmi_watchdog action=power_cycle panic_wdt_timeout=10
NOTE: reboot or reload ipmi_watchdog module to take the changes in effect.
Dell IDrac (module "ipmi_watchdog")
For Dell IDrac, please desactivate the Automated System Recovery Agent in IDrac configuration.
If openmanage is installed, you need to disable watchdog management from openmanage
/opt/dell/srvadmin/sbin/dcecfg command=removepopalias aliasname=dcifru
and reboot server
After restart, check that watchdog timer is 10s, and not overrided by openmanage
idracadm getsysinfo -w Watchdog Information: Recovery Action = Power Cycle Present countdown value = 9 seconds Initial countdown value = 10 seconds
or
# ipmitool mc watchdog get Watchdog Timer Use: SMS/OS (0x44) Watchdog Timer Is: Started/Running Watchdog Timer Actions: Hard Reset (0x01) Pre-timeout interval: 0 seconds Timer Expiration Flags: 0x00 Initial Countdown: 10 sec Present Countdown: 9 sec
HP ILO (module "hpwdt" )
Users have reported crash with this module, please test and update the wiki if it's working fine
also, disable HP ASR feature (Automatic Server Recovery).
http://h17007.www1.hp.com/docs/iss/proliant_uefi/s_asr_status.html
If you have installed hp management tools, you need to disable "hp-asrd" daemon
Troubleshooting
Error recovery
If a service start fails we try to recover from it with our "restart" and "relocate" policy, see the man page of the ha-manager for more information.
If after all tries the service state could not be recovered it gets placed in an error state. In this state the service won't get touched by the HA stack anymore. To recover from this state you should follow these steps:
- bring the resource back into an safe and consistent state (e.g: killing its process)
- disable the ha resource to place it in an stopped state
- fix the error which led to this failures
- after you fixed all errors you may enable the service again
Note: when a Service fails to stop it also get's placed in the error state, you may follow the same steps to recover from it.
Failed watchdog-mux or Multiple Watchdogs
Disable all BIOS watchdog functionality, those settings setup the watchdog in the expectancy that the OS resets it, that is not our desired use case here and may lead to problems - e.g.: reset of the node after a fixed amount of time.
Intel AMT (OS Health Watchdog) should be disabled and with it the mei and mei_me modules, as they may cause problems.
If you host has multiple watchdogs available, only allow the one you want to use for HA, i.e. blacklist the other modules from loading. Our watchdog multiplexer will use /dev/watchdog which maps to /dev/watchdog0.
Selecting a specific watchdog is not implemented, mainly for this quote from the linux-watchdog mailing list:[1]
The watchdog device node <-> driver mapping is fragile and can change from one kernel version to the next or even across reboot, so users shouldn't assume it to be persistent.
Deleting Nodes From The Cluster
When deleting a node from a HA cluster you have to ensure the following:
- all HA services were relocate to another node! A graceful shutdown will NOT auto migrate them.
- Remove the node from all defined groups.
- Shutdown the node you want to remove, from now on this node MUST NOT come online in the same network again, without being reinstalled/cleared of all cluster traces.
- execute `pvecm delnode nodename` from an remaining node.
The HA stack now places the node in an 'gone' state, you still see it in the manager status. After an hour in this state it will be auto deleted. This ensures that if the node died ungracefully the services still will be fenced and migrated to another node.
Durations
Note that some HA actions may take their time, and don't happen instantly. This avoids out of control feedback loops, and ensures that the HA stack is in a safe and consistent state all the time.
Container
Note that while containers may be put under HA, currently (PVE 4.1) they don't support live migration. So all migrate actions on them will be mapped to relocate (stop, move, start). If a node fails recovery works after the failed node was fenced, as long as you don't use local bound resources.
Video Tutorials
Testing
Before going into production it is highly recommended to do as many tests as possible. Then, do some more.
Useful command line tools
Here is a list of useful CLI tools:
- ha-manger - to manage the ha stack of the cluster
- pvecm - to manage the cluster-manager
- corosync* - to manipulate the corosync