Sheepdog cluster install

From Proxmox VE
Jump to: navigation, search


Introduction

In this howTo we will setup a Sheepdog Cluster, using an interactive script for the installation/configuration.

NOTE: although is experimental and not officially supported on Proxmox VE, sheepdog has now a commercial support.

Download the script

These scripts will help in the installation and in the configuration of the sheepdog node, we need git to download them.

EDIT : link changed

#git clone https://github.com/sheepdog/sheepdog-utils

Install zookeeper and sheep

Launch the script:

cd sheepdog-utils/
./install.sh

So the script will print out:


If you already installed the standard qemu-kvm package it will be removed.
If /usr/src/qemu or /usr/src/sheepdog exist, they will be replaced after trying
to uninstalling them.

Do you whant do proceed?

Chosse what you need to install

1) sheepdog
2) sheepdog + qemu
3) sheepdog + qemu + zookeeper
4) help

Number:

Now you can choose:

  • 1 it will install only sheepdog.
  • 2 it will install sheepdog and qemu.
  • 3 it will install sheepdog, qemu and zookeeper.

Zookeeper is the alternative daemon to corosync. For sheepdog, it way work better and you need to install it only on 3 nodes(*) for managing a cluster of hundreds of sheepdog nodes. Obviously, for bigger clusters we appeal to the common sense, 3 zookeeper node is a bit dangerous if you lose one of them.

(*) Zookeeper need at least 3 node, it's possible to create a 2 node cluster, but in case of the loss of one node, you'll experience the split brain problem.

So we follow the more complex choice, installing sheepdog+qemu+zookeeper

After downloading all the necessary it will ask:

More than one ip has beed deteceted:
192.168.2.59 192.168.70.100
Type the ip that will be used by zookeeper.
ip:

Here you have to put the ip address that zookeper should use, then:

Type the other zookeeper nodes ip, separated by a single space.
If this is the only zookeeper node, just press enter.
ip: 

Here you have to list all the ip of the nodes that will have zookeper installed. Keep in mind that you need at least 3 zookeper nodes. You can add the ip address later on editing the zookeper configuration file. ( /etc/zookeeper/conf/zoo.cfg )

Next step it will set the right fd limit ( max number of open files ) and ask for a reboot. After this you should run:

./assistant.sh

This script will configure sheepdog to work with the disks and partitions you like.

So the script now will ask as follow:

Before running this script:
- configure all your network cards
- edit your fstab and mount all the required devices

Press ctrl+c sto stop it at anytime.
Continue?

(y/n): 

In the fstab you should need to prepare something like this:

/dev/sda1 /mnt/sheep/disk01 xfs defaults,noatime        0       0
/dev/sda1 /mnt/sheep/disk02 xfs defaults,noatime        0       0
/dev/sda1 /mnt/sheep/disk03 xfs defaults,noatime        0       0

We use xfs but you can choose any filesystem that could support xattr. Press yes if your setting are done.

Is this node going to be a gateway only?                                                                                                       
(A gateway only node is able to run guests, but it doesn't store any data.                                                                     
The cluster size is not going to be affected).                                                                                                 
                                                                                                                                               
(y/n):                                   

Here if you want to use the node as gateway only or as storage. We choose No.

Remember that each device must have extended attributes enabled.                                                                               
Check your /etc/fstab if in doubt.                                                                                                             
The following mount point has been detected:                                                                                                   
                                                                                                                                               
/mnt/sheep/disk01                                                                                                                                      
/mnt/sheep/disk02
/mnt/sheep/disk03                                                                                                                                
                                                                                                                                               
Type the mount point(s) used by sheepdog, one per line.                                                                                        
(Leave blank once done)                                                                                                                        
path:

So put the /mnt/sheep/disk01  and press enter
Do it for all the partition you like to use in the cluster.

The selected mount point are: /mnt/sheep/disk01 /mnt/sheep/disk02 /mnt/sheep/disk03                                        
                                                                                                                                                                                                                                                                    
This node has been recognized as a Zookeeper node.
These servers' ip will be used:

192.168.2.59
192.168.2.60

Are these the right zookeeper' ip(s)?

(y/n):

Type yes.


SUMMARY
=======
Gateway only:   no
Devices:        /mnt/old_boot
Dedicate nic:
Zookeeper:      /mnt/sheep/disk01 /mnt/sheep/disk02 /mnt/sheep/disk03
=======

If everythin is correct, press 'y' or 'n' to exit

(y/n): 


Yes 

Now you should have a correctly configured sheepdog node. Just repeat the install.sh and assistant.sh for all the node you need.

Contributor and support

If you need support, contact Extensys S.R.L.