From Proxmox VE

Introduction

Intel Modular Server (IMS) is a powerful server platform with integrated SAN. This how to describes the installation and configuration of Proxmox VE on IMS - with the support of live migration (KVM guests and containers) between compute modules.

A big thanks to Intel, sponsoring the hardware to the Proxmox VE test lab.

Before you start

Make sure you got the latest firmware on the IMS - at the time of these tests we got the firmware V5.5.

Depending on the workload, you need to clarify among all other things the following in advance:

Number of KVM guests, including storage requirements
Number of OpenVZ containers, including storage requirements
Backup considerations

Hardware configuration

Storage

The storage drive bay supports six 3.5" SAS/SATA or fourteen 2.5" SAS hard disk drives. We prefer the model with 14 drives as it provides more flexibility.

Note:

If you need more storage (more than 14 disks), you can connect external storage via mini SAS port, or you can connect your existing iSCSI or NFS storage.

Storage is the most important part for overall performance. So if your budget is limited, use cheaper CPU´s but never cheap and slow hard drives.

Pool1: Proxmox VE installation disk (and container storage)

Create a storage pool for all installation disks (for best performance, use 4 disks)
Create virtual drives (for best performance, use Raid10), and assign each one to a compute module as the boot drive (0)

Keep in mind, that all OpenVZ containers are also stored on these disks.

Example: Four 300GB disks and three compute modules

As we use Raid10, we have a total capacity of 600 GB, so we can assign 200 GB to each compute module for the installation and the local storage (mainly used to store containers).

Pool2: Shared LVM storage for KVM guests

By using the shared LUN feature, you can assign a virtual disk to multiple compute modules enabling live migration of KVM guests between compute modules.

This is one of the main advantages of the IMS, as you do not need an extra SAN box. No need for expensive fiber channel (FC) infrastructure or the complexity of configuration of an extra storage network for iSCSI.

Create a storage pool for LVM storage (for best performance, use 4 disks)
Create one virtual drive (for best performance, you can also use Raid10) and assign the disk to ALL compute modules as drive 1.

Example: Four 300GB disks

As we use Raid10, we have a total capacity of 600 GB, so we can use 600 GB for our KVM virtual machines.

Pool3: Backup and ISO images - NFS

Create a storage pool, big enough for storing the backup data and ISO images
Create one big virtual drive and assign it to one single compute module as the third drive (2)

Example: Three 300GB disks

We use Raid5, so we have 600 GB for backups and ISO images.

Hot spare drive

We recommend you add at least one global hot spare disk. We suggest to use identical disks on all slots.

Compute Modules

You can use up to six compute modules each one with 2 Intel Xeon processors 5500 series´s and up to 96 GB of memory.

Our test setup:

Three MFS5520VI
Each module has two Xeon E5540
and 12 GB memory (6 x 2 GB)

Network

Nothing special needed.

Installation of Proxmox VE

Bare-metal ISO install

By using the integrated remote console, installation can be done by mapping the ISO images to the compute modules. Note: it is recommended to install via USB Stick as the iso redirect does not always work and is slower. See also Installation After reboot, install the latest packages on all nodes:

aptitude update && aptitude safe-upgrade

Proxmox VE package list:

pveversion -v

pve-manager: 1.5-8 (pve-manager/1.5/4674)
running kernel: 2.6.18-2-pve
proxmox-ve-2.6.18: 1.5-5
pve-kernel-2.6.18-2-pve: 2.6.18-5
qemu-server: 1.1-11
pve-firmware: 1.0-3
libpve-storage-perl: 1.0-10
vncterm: 0.9-2
vzctl: 3.0.23-1pve8
vzdump: 1.2-5
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm-2.6.18: 0.9.1-5

Create Cluster

Promote one server to the master and join the other nodes, see Proxmox VE Cluster

Setup the LVM disk for KVM guest

Login to the master via SSH and partition the disk:

Create a big single partition (sdb1)

fdisk /dev/sdb

Create the physical volume:

pvcreate /dev/sdb1
 Physical volume "/dev/sdb1" successfully created

Create the volume group 'vmdisks' (just choose a unique name):

vgcreate vmdisks /dev/sdb1
  Volume group "vmdisks" successfully created

And finally: Add the LVM Group to the storage list via the web interface.

Optional: Setup a NFS server on one Proxmox VE node

For small setups its quite useful to install a NFS server just on Proxmox VE (If you already have a NFS server in your network, just use the existing one).

Login to the server where the third hard disk is assigned via SSH and partition the disk:

Create a big single partition (sdc1)

fdisk /dev/sdc

Format the partition with ext3

mkfs.ext3 /dev/sdc1

query the UUID of the partition

blkid

Add the partition to your /etc/fstab (replace the UUID with yours)

nano /etc/fstab

UUID=2f1fe410-293f-4f1d-bdfb-8b45f1d3b4aa /srv ext3 defaults 0 2

create dir´s:

mkdir /srv
cd /srv
mkdir iso
mkdir backup

install nfs-server package (use the packages from lenny-backports, nfs-kernel-server >= 1.2.2-1)

aptitude -t lenny-backports install nfs-kernel-server

Configure nfs exports (just an example, adapt this to your network setup):

nano /etc/exports
/srv/backup     192.168.0.0/255.255.240.0(rw,sync,no_subtree_check,no_root_squash)
/srv/iso        192.168.0.0/255.255.240.0(rw,sync,no_subtree_check,no_root_squash)

And finally, restart NFS server:

/etc/init.d/nfs-kernel-server restart

NFS-ISO Store

Now, you can add the NFS-ISO Store via web interface.

Backup Store

Add the NFS-Backup Store via web interface.

Optional: Configure Multi-Path

The Intel modular server can be configured with two storage controllers. If one controller goes down the other one can take over. In order for fail-over to work the OS on each blade must be configured to be multipath aware. The ProxMox 2.x bare-metal iso, based on Debian 6, is not fully multi-path aware by default. The following is a how-to tested on ProxMox VE 2.2. Many thanks to Tozz, Kaya, mjoconr and others on the forum for help with this! This how-to is based on the above flex server configuration with installation disk = sda1 shared LVM storage = sdb1 and backup/iso images = sdc1

Install the following packages:

aptitude install multipath-tools multipath-tools-boot libfuse2 liblzma2

The grub packages in Debian 6 do not support multipath-tools-boot and therefore you need to install packages from Debian 7 repositories. The following package links worked on February 18th 2013, but may need to be updated as packages change, download and install the following packages: (you can just cut and paste these in the command land to download, use dpkg -i to install)

wget http://ftp.us.debian.org/debian/pool/main/e/eglibc/multiarch-support_2.13-38_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/x/xz-utils/liblzma5_5.1.1alpha+20120614-2_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grub/grub-legacy_0.97-67_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grub2/grub-pc_2.00-13_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grub2/grub-common_2.00-13_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grub2/grub2-common_2.00-13_amd64.deb
wget http://ftp.us.debian.org/debian/pool/main/g/grub2/grub-pc-bin_2.00-13_amd64.deb

Find the wwid of each disk

# /lib/udev/scsi_id --whitelisted --device=/dev/sda
# /lib/udev/scsi_id --whitelisted --device=/dev/sdb
# /lib/udev/scsi_id --whitelisted --device=/dev/sdc

create /etc/multipath.conf and add the wwid's and aliases for each disk. In the example below I have added an entry for the installation disk sda = system, shared LVM Storage sdb = vmdisks and backup/iso images sdc = vzdumps.

blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(hd|xvd)[a-z][[0-9]*]"
        devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
}
devices {
        device {
        vendor                  "Intel"
        product                 "Multi-Flex"
        path_grouping_policy    "group_by_prio"
        getuid_callout          "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        prio                    "alua"
        path_checker            tur
        path_selector           "round-robin 0"
        hardware_handler        "1 alua"
        failback                immediate
        rr_weight               uniform
        rr_min_io               100
        no_path_retry           queue
        features                "1 queue_if_no_path"
        }
}


multipaths {         
        multipath {
                 wwid    id-of-sda
                 alias   system
         }
        multipath {
                 wwid    id-of-sdb
                 alias   vmdisks
         }
        multipath {
                 wwid    id-of-sdc
                 alias  vzdumps
         }

}

Restart multipath

/etc/init.d/multipath-tools-boot restart; /etc/init.d/multipath-tools restart

Modify /etc/fstab for new multipath devices (aliases) This is an example of my /etc/fstab, in this case my two drives are listed by UUID but they could also be listed as /dev/sda and /dev/sdc. I commented the old entries out so you can see what it looked like before the change. Note: there is no entry for vmdisks or /dev/sdb

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1

#UUID=294b7d72-4dea-4549-97c9-4320a55f3fb5 /boot ext3 defaults 0 1
/dev/mapper/system-part1 /boot ext3 defaults 0 1

/dev/pve/swap none swap sw 0 0

proc /proc proc defaults 0 0

#UUID=777b58c2-c780-41c3-919f-3640e4f5cce6 /srv ext3 defaults 0 2
/dev/mapper/vzdumps-part1 /srv ext3 defaults 0 2

Re-create inittramfs

update-initramfs -u

Disable UUID in /etc/default/grub
update Grub

update-grub

Reboot Server and press ESC during GRUB. Edit boot path by replacing UUID with /dev/mapper/system-part1 and boot system by hitting F10
After the system boots up, log in and do a "update-grub" to save the new boot path. Reboot the server to make sure all is working.
you can see if mulitpath is working with the following command:

multipath -ll

the output should look something like this:

vmdisks (22226000155e33228) dm-1 Intel,Multi-Flex
size=1.1T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:1 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 0:0:1:1 sde 8:64 active ready running
system (222980001557edea5) dm-0 Intel,Multi-Flex
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:0 sda 8:0 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 0:0:1:0 sdd 8:48 active ready running
vzdumps (222dd000155416965) dm-2 Intel,Multi-Flex
size=557G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 0:0:0:2 sdc 8:32 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 0:0:1:2 sdf 8:80 active ready running

Then for the final test: pull out the storage controller to which the blade you just configured has affinity and see if your system still works. Don't do this on a production system!

Install Virtual Machines

Get templates

In order to create a container, you need to get templates first - Either download them via web interface or just upload via web interface (if you got them already). You can also transfer templates directly via scp to /var/lib/vz/template/cache/ on the master server (these templates will be replicated to all cluster nodes).

Upload ISO images to the NFS-ISO Store

In order to install KVM guests, you need a installation media - we use ISO images for maximum convenience. Upload them via web interface (DVD´s cannot be uploaded, size limit) or transfer them via scp to /mnt/pve/NFS-ISO/.

Links

Personal tools

Intel Modular Server