Proxmox VE Cluster Version 1 *OLD*

From Proxmox VE
(Redirected from Proxmox VE Cluster)
Jump to: navigation, search
Yellowpin.svg Note: Article about the outdated Proxmox VE 1.x releases
(thumbnail)
Status of 3 Cluster nodes

Contents

Introduction

Proxmox VE Cluster enables central management of multiple physical servers. A Proxmox VE Cluster consists of one master and several nodes (minimum is a master and one node).

Main features

  • Centralized web management
  • One login and password for accessing all nodes and guests
  • Console view to all Virtual Machines
  • Migration of Virtual Machines between physical hosts
  • Synchronized Virtual Appliance template store

Create a Proxmox VE Cluster

First, install two Proxmox VE servers, see Installation. Make sure that each Proxmox VE server has a unique host name, by default all server has the same host name.

Currently the cluster creation has to be done on the console, you can login to the Proxmox VE server via ssh.

All settings can be done via "pveca", the PVE Cluster Administration Toolkit

USAGE: pveca -l             # show cluster status
       pveca -c             # create new cluster with localhost as master
       pveca -s [-h IP]     # sync cluster configuration from master (or IP)
       pveca -d ID          # delete a node
       pveca -a [-h IP]     # add new node to cluster
       pveca -m             # force local node to become master

Define the master

Login via ssh to the first Proxmox VE server.

Create the master:

pveca -c

To check the state of cluster:

pveca -l

Add a node to an existing master

Login via ssh to a second Proxmox VE server. Please note, the node should not have any VM´s. (If yes you will get conflicts with identical VMID´s - to workaround, use vzdump to backup and to restore to a different VMID after the cluster configuration).

Join a node to the master:

pveca -a -h IP-ADDRESS-MASTER

To check the state of cluster:

pveca -l

Display the state of cluster:

pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
 1 : 192.168.7.104   M     A    5 days 01:43   0.54    20%     1%     4%
 2 : 192.168.7.103   N     A    2 days 05:02   0.04    26%     5%    29%
 3 : 192.168.7.105   N     A           00:13   1.41    22%     3%    15%
 4 : 192.168.7.106   N     A           00:05   0.54    17%     3%     3%

Remove a cluster node

Through the Central Web-based Management move all virtual machines out of the node you need to remove from the cluster. Make sure you have no local backups you want to keep, and save them accordingly.

Log into the cluster master node via ssh. Issue a pceva command to identify the nodeID:

pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
 1 : 172.25.3.40     M     A   58 days 00:02   0.02     7%    75%
 2 : 172.25.3.41     N     A   57 days 22:42   0.02     2%    90%
 3 : 172.25.3.39     N     A   59 days 00:26   0.00     2%    14%

Issue the delete command (here deleting node #3):

pveca -d 3

If the operation succeeds no output is returned.

PVE cluster unconfigures the slave node and shortly it will not be visible into the Central Web-based Management.

Please note that the node will not see the shared storage anymore.

Upgrade a working cluster

NOTE: if you got a critical production system you should test upgrades in your test lab.

Here are two possible ways to perform cluster upgrades (new pve versions, etc), example for a simple two nodes cluster:

method A

If you want to keep the disruption to each vm as short as possible:

  1. Live Migrate all VMs to the master node
  2. Update the slave node.
  3. Reboot Slave node.
  4. One by One turn off a VM, offline migrate to updated node, turn it on on new node.
  5. After all vms are moved, update the master node and reboot it.
  6. Then live migrate some of the vms back to the master node where they belong.

This can also be done backwards, i.e update the master first and then the slave. Users report to have done this without issues. But it could be better to update the slaves first then the master.

method B

Users have also updated like this:

  1. turn off all the vms on a node.
  2. install updates
  3. reboot
  4. let VMs autostart or manually start them
  5. repeat process till all nodes are updated.

see also this forum thread and Upgrade and Downloads wiki pages

Working with Proxmox VE Cluster

Now, you can start creating Virtual Machines on Cluster nodes by using the Central Web-based Management on the master.

Troubleshooting

You can manually check the cluster configuration files on each node. Before you edit these files, stop the cluster sync and tunnel service via web interface.

nano /etc/pve/cluster.cfg

Also check if the following file are still up to date, if not because your keys got updated just remove/adapt it.

/root/.ssh/known_hosts

Delete and recreate a cluster configuration

Sometimes it's quicker to delete and recreate your cluster configuration than it is to try and figure out what went wrong. The process includes stopping the cluster sync and tunnel service, deleting the existing cluster config and creating/joining the new cluster. The steps can be found below:

  • Note: If your ssh host keys have changed you may need to delete them on each host before you begin:
rm /root/.ssh/known_hosts
  • Jot down the IP of the new master node.
  • Run on new master node:
/etc/init.d/pvemirror stop
/etc/init.d/pvetunnel stop
rm /etc/pve/cluster.cfg
pveca -c
  • Verify that the cluster has been created:
pveca -l
  • Run on nodes you wish to add to the new cluster.
    • Note: Change IP-ADDRESS-MASTER to the IP of the new master node.
/etc/init.d/pvemirror stop
/etc/init.d/pvetunnel stop
rm /etc/pve/cluster.cfg
pveca -a -h IP-ADDRESS-MASTER
  • Verify that the node has been added to the new cluster:
pveca -l

Local node already part of cluster

If you are getting the error "local node already part of cluster" and you have verified that the host name is different than any other node then you may have not properly indented you /etc/network/interfaces file. Example:

auto vmbr0
iface vmbr0 inet static
address 10.9.0.2
netmask 255.255.255.0
gateway 10.9.0.1
bridge_ports bond0.9
bridge_stp off
bridge_fd 0

Should be:

auto vmbr0
iface vmbr0 inet static
        address 10.9.0.2
        netmask 255.255.255.0
        gateway 10.9.0.1
        bridge_ports bond0.9
        bridge_stp off
        bridge_fd 0

Video Tutorials

Tutorials

Personal tools
Namespaces

Variants
Actions
Navigation
Virtual Appliances
Sites
Tools