Storage: Ceph
From Proxmox VE
Contents |
Build a ceph storage cluster
Currently, it's not possible to build the cluster on proxmox host, so you need to use others servers.
You need 3 servers minimum.
Proxmox Support CEPH >= 0.56
Recommandation is to use ubuntu 12.04 lts as distro os, this is the distro used by intank for the ceph development. (you need recent filesystem version and glibc)
Official installation documentation is here : http://ceph.com/docs/master/install/
On each server :
- Install ubuntu.
- add ceph repositories
wget -q -O- https://raw.github.com/ceph/ceph/master/keys/release.asc | sudo apt-key add - echo deb http://ceph.com/debian/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list
- install packages
sudo apt-get update && sudo apt-get install ceph
-create a /etc/ceph/ceph.conf file
This exemple show 3 servers:
- Each server have 5 disks for storage and 1 ssd disk for journal
- For each disk, you need 1 osd daemon.
- Each server has also a monitor daemon
[global]
; use cephx or none
auth supported = cephx
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
[mon]
[mds]
[osd]
osd journal = /srv/journal/osd.$id.journal ;only if you want put journal on a dedicated disk, like ssd
osd journal size = 1000
; uncomment the following line if you are mounting with ext4
; filestore xattr use omap = true
; uncomment if you want journal on tmpfs
;journal dio = false
[mon.a]
host = server1
mon addr = 192.168.0.1:6789
[mon.b]
host = server2
mon addr = 192.168.0.2:6789
[mon.c]
host = server3
mon addr = 192.168.0.3:6789
[osd.0]
host = server1
addr = 192.168.0.1
[osd.1]
host = server1
addr = 192.168.0.1
[osd.2]
host = server1
addr = 192.168.0.1
[osd.3]
host = server1
addr = 192.168.0.1
[osd.4]
host = server1
addr = 192.168.0.1
[osd.5]
host = server2
addr = 192.168.0.2
[osd.6]
host = server2
addr = 192.168.0.2
[osd.7]
host = server2
addr = 192.168.0.2
[osd.8]
host = server2
addr = 192.168.0.2
[osd.9]
host = server2
addr = 192.168.0.2
[osd.10]
host = server3
addr = 192.168.0.3
[osd.11]
host = server3
addr = 192.168.0.3
[osd.12]
host = server3
addr = 192.168.0.3
[osd.13]
host = server3
addr = 192.168.0.3
[osd.14]
host = server3
addr = 192.168.0.3
Then you need to format each disk, with xfs or btrfs. XFS is the recommendation for now.
on each server
mkfs.xfs /dev/sdb mkfs.xfs /dev/sdc mkfs.xfs /dev/sdd mkfs.xfs /dev/sde mkfs.xfs /dev/sdf mkfs.xfs /dev/sdg
server1 :
mkdir /var/lib/ceph/osd/ceph-0/ mkdir /var/lib/ceph/osd/ceph-1/ mkdir /var/lib/ceph/osd/ceph-2/ mkdir /var/lib/ceph/osd/ceph-3/ mkdir /var/lib/ceph/osd/ceph-4/ mkdir /var/lib/ceph/mon/ceph-a/
/etc/fstab configuration:
/dev/sdb /var/lib/ceph/osd/ceph-0 xfs noatime,nodiratime 0 0 /dev/sdc /var/lib/ceph/osd/ceph-1 xfs noatime,nodiratime 0 0 /dev/sdd /var/lib/ceph/osd/ceph-2 xfs noatime,nodiratime 0 0 /dev/sde /var/lib/ceph/osd/ceph-3 xfs noatime,nodiratime 0 0 /dev/sdf /var/lib/ceph/osd/ceph-4 xfs noatime,nodiratime 0 0 /dev/sdg /srv/journal xfs noatime,nodiratime 0 0
server2 :
mkdir /var/lib/ceph/osd/ceph-5/ mkdir /var/lib/ceph/osd/ceph-6/ mkdir /var/lib/ceph/osd/ceph-7/ mkdir /var/lib/ceph/osd/ceph-8/ mkdir /var/lib/ceph/osd/ceph-9/ mkdir /var/lib/ceph/mon/ceph-b/
/etc/fstab configuration:
/dev/sdb /var/lib/ceph/osd/ceph-5 xfs noatime,nodiratime 0 0 /dev/sdc /var/lib/ceph/osd/ceph-6 xfs noatime,nodiratime 0 0 /dev/sdd /var/lib/ceph/osd/ceph-7 xfs noatime,nodiratime 0 0 /dev/sde /var/lib/ceph/osd/ceph-8 xfs noatime,nodiratime 0 0 /dev/sdf /var/lib/ceph/osd/ceph-9 xfs noatime,nodiratime 0 0 /dev/sdg /srv/journal xfs noatime,nodiratime 0 0
server3 :
mkdir /var/lib/ceph/osd/ceph-10/ mkdir /var/lib/ceph/osd/ceph-11/ mkdir /var/lib/ceph/osd/ceph-12/ mkdir /var/lib/ceph/osd/ceph-13/ mkdir /var/lib/ceph/osd/ceph-14/ mkdir /var/lib/ceph/mon/ceph-c/
/etc/fstab configuration:
/dev/sdb /var/lib/ceph/osd/ceph-10 xfs noatime,nodiratime 0 0 /dev/sdc /var/lib/ceph/osd/ceph-11 xfs noatime,nodiratime 0 0 /dev/sdd /var/lib/ceph/osd/ceph-12 xfs noatime,nodiratime 0 0 /dev/sde /var/lib/ceph/osd/ceph-13 xfs noatime,nodiratime 0 0 /dev/sdf /var/lib/ceph/osd/ceph-14 xfs noatime,nodiratime 0 0 /dev/sdg /srv/journal xfs noatime,nodiratime 0 0
then mount each disk on each server
mount /dev/sdb mount /dev/sdc mount /dev/sdd mount /dev/sde mount /dev/sdf mount /dev/sdg
on First server
Generate a ssh key
ssh-keygen -t rsa
And copy it on the 2 others server
scp .ssh/id_rsa.pub server2:.ssh/authorized_keys2 scp .ssh/id_rsa.pub server3:.ssh/authorized_keys2
Create the ceph cluster
cd /etc/ceph mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring
Start your cluster
/etc/init.d/ceph -a start
Your cluster is ready !
Configure Proxmox to use the ceph cluster
GUI
You can use proxmox GUI to add the rbd storage
Manual configuration edit
edit your /etc/pve/storage.cfg and add the configuration
rbd: mycephcluster
monhost 192.168.0.1:6789;192.168.0.2:6789;192.168.0.3:6789
pool rbd (optionnal, default =r rbd)
username admin (optionnal, default = admin)
content images
note: you must use ip (no dns fqdn) for monhost
Authenfication
If you use cephx authentification,you need to copy the keyfile from ceph to proxmox ve host
scp cephserver1:/etc/ceph/ceph.keyring /etc/pve/priv/ceph/mycephcluster.keyring
That's all !
you can now create your disk on the ceph storage :)
