Ceph Server: Difference between revisions

From Proxmox VE
Jump to navigation Jump to search
No edit summary
 
(43 intermediate revisions by 10 users not shown)
Line 1: Line 1:
== Introduction ==
<div class="warn-box">This article got replaced by our reference documentation's ceph chapter: https://pve.proxmox.com/pve-docs/chapter-pveceph.html</div>
 
[[Image:Screen-Ceph-Status.png|thumb]] Ceph is a distributed object store and file system designed to provide excellent performance, reliability and scalability - See more at: http://ceph.com.
 
Proxmox VE supports Ceph’s RADOS Block Device to be used for VM and container disks. The Ceph storage services are usually hosted on external, dedicated storage nodes. Such storage clusters can sum up to several hundreds of nodes, providing petabytes of storage capacity.
 
For smaller deployments, it is also possible to run Ceph services directly on your Proxmox VE nodes. Recent hardware has plenty of CPU power and RAM, so running storage services and VM/CTs on same node is possible.
 
This articles describes how to setup and run Ceph storage services directly on Proxmox VE nodes. If you want to install and configure an external Ceph storage read the [http://docs.ceph.com/docs/master/ Ceph documentation]. To configure an external Ceph storage works as described in section [[#Ceph Client | Ceph Client]] accordingly.
 
== Advantages ==
 
*Easy setup and management with CLI and GUI support on Proxmox VE
*Thin provisioning
*Snapshots support
*Self healing
*No single point of failure
*Scalable to the exabyte level
*Setup pools with different performance and redundancy characteristics
*Data is replicated, making it fault tolerant
*Runs on economical commodity hardware
*No need for hardware RAID controllers
*Easy management
*Open source


== Recommended hardware ==
== Recommended hardware ==
Take a look at our Proxmox VE Ceph Benchmark 2018/02 for possible hardware decisions.
https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark


You need at least three identical servers for the redundant setup. Here is the specifications of one of our test lab clusters with Proxmox VE and Ceph (three nodes):
== Further readings about Ceph ==
 
*Dual Xeon E5-2620v2, 64 GB RAM, Intel S2600CP mainboard, Intel RMM, Chenbro 2U chassis with eight 3.5” hot-swap drive bays, 2 fixed 2.5" SSD bays
*10 GBit network for Ceph traffic (one Dual 10 Gbit Intel X540-T2 in each server, one 10Gb switch - Cisco SG350XG-2F1)
*Single enterprise class SSD for the Proxmox VE installation (because we run Ceph monitors there and quite a lot of logs), we use one Samsung SM863 240 GB per host.
*Use at least two SSD as OSD drives. You need high quality and enterprise class SSDs here, never use consumer or "PRO" consumer SSDs. In our testsetup, we have 4 Intel SSD DC S3520 1.2TB, 2.5" SATA SSD per host for storing the data (OSD, no extra journal) - This setup delivers about 14 TB storage. By using the redundancy of 3, you can store up to 4,7 TB (100%). But to be prepared for failed disks and hosts, you should never fill up your storage with 100&nbsp;%.
*As a general rule, the more OSD the better, fast CPU (high GHZ) is also recommended. NVMe express cards are also possible, e.g. mix of slow SATA disks with SSD/NVMe journal devices.
 
Again, if you expect good performance, always use enterprise class SSD only, we have good results in our testlabs with:
*SATA SSDs:
**Intel SSD DC S3520
**Intel SSD DC S3610
**Intel SSD DC S3700/S3710
**Samsung SSD SM863
*NVMe PCIe 3.0 x4 as journal:
**Intel SSD DC P3700
 
By adding more OSD SSD/disks into the free drive bays, the storage can be expanded. Of course, you can add more servers too as soon as your business is growing, without service interruption and with minimal configuration changes.
 
If you do not want to run virtual machines and Ceph on the same host, you can just add more Proxmox VE nodes and use these for running the guests and the others just for the storage.
 
== Installation of Proxmox VE ==
 
Before you start with Ceph, you need a working Proxmox VE cluster with 3 nodes (or more). We install Proxmox VE on a fast and reliable enterprise class SSD, so we can use all bays for OSD (Object Storage Devices) data. Just follow the well known instructions on [[Installation]] and [[Proxmox VE 2.0 Cluster]].
 
'''Note:'''
 
Use ext4 if you install on SSD (at the boot prompt of the installation ISO you can specify parameters, e.g. "linux ext4 swapsize=4").
 
== Network for Ceph ==
 
All nodes need access to a separate 10Gb network interface, exclusively used for Ceph. We use network 10.10.10.0/24 for this tutorial.
 
It is highly recommended to use 10Gb for that network to avoid performance problems. Bonding can be used to increase availability.
 
If you do not have fast network switches, you can also setup a [[Full Mesh Network for Ceph Server]].
 
=== First node ===
 
The network setup (ceph private network) from our first node contains:
 
# from /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  10.10.10.1
  netmask  255.255.255.0
 
=== Second node ===
 
The network setup (ceph private network) from our second node contains:
 
# from /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  10.10.10.2
  netmask  255.255.255.0
 
=== Third node ===
 
The network setup (ceph private network) from our third node contains:
 
# from /etc/network/interfaces
auto eth2
iface eth2 inet static
  address  10.10.10.3
  netmask  255.255.255.0
 
== Installation of Ceph packages ==
 
You now need to select 3 nodes and install the Ceph software packages there. We wrote a small command line utility called 'pveceph' which helps you performing this tasks, you can also choose the version of Ceph. Login to all your nodes and execute the following on all:
 
node1# pveceph install -version jewel
 
node2# pveceph install -version jewel
 
node3# pveceph install -version jewel
 
This sets up an 'apt' package repository in /etc/apt/sources.list.d/ceph.list and installs the required software.
 
== Create initial Ceph configuration ==
 
[[Image:Screen-Ceph-Config.png|thumb]] After installation of packages, you need to create an initial Ceph configuration on just one node, based on your private network:
 
node1# pveceph init --network 10.10.10.0/24
 
This creates an initial config at /etc/pve/ceph.conf. That file is automatically distributed to all Proxmox VE nodes by using [http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_%28pmxcfs%29 pmxcfs]. The command also creates a symbolic link from /etc/ceph/ceph.conf pointing to that file. So you can simply run Ceph commands without the need to specify a configuration file.
 
== Creating Ceph Monitors ==
 
[[Image:Screen-Ceph-Monitor.png|thumb]] After that you can create the first Ceph monitor service using:
 
node1# pveceph createmon
 
== Continue with CLI or GUI ==
 
As soon as you have created the first monitor, you can start using the Proxmox GUI (see the video tutorial on [http://youtu.be/ImyRUyMBrwo Managing Ceph Server]) to manage and view your Ceph configuration.
 
Of course, you can continue to use the command line tools (CLI). &nbsp;We continue with the CLI in this wiki article, but you should achieve the same results no matter which way you finish the remaining steps.
 
== Creating more Ceph Monitors ==
 
You should run 3 monitors, one on each node. Create them via GUI or via CLI. So please login to the next node and run:
 
node2# pveceph createmon
 
And execute the same steps on the third node:
 
node3# pveceph createmon
 
'''Note:'''
 
If you add a node where you do not want to run a Ceph monitor, e.g. another node for OSDs, you need to install the Ceph packages with 'pveceph install' and you need to initialize Ceph with 'pveceph init'
 
PS: ''' per thread https://forum.proxmox.com/threads/ceph-nodes-alternate-red-green-on-pve-screen.32670/  as of 2017-02-06 a node will seem to have issues [ really not ] on pve screen.  red/green alternating . so as of this writing you must do this for any node that is not a ceph mon:'''
pveceph init


== Creating Ceph OSDs ==
Ceph comes with plenty of documentation [https://docs.ceph.com/ here]. Even better, the dissertation from the creator of Ceph - Sage A. Weil - is also [http://unixadmin.free.fr/wp-content/uploads/weil-thesis.pdf available]. By reading that thesis you can get an in-depth insight how Ceph works.


[[Image:Screen-Disks.png|thumb]] [[Image:Screen-Ceph-OSD-Status.png|thumb]] First, please be careful when you initialize your OSD disks, because it basically removes all existing data from those disks. So it is important to select the correct device names. The Proxmox VE GUI displays a list of all disk, together with device names, usage information and serial numbers.  
* https://ceph.com/


Creating OSDs can be done via GUI - self explaining - or via CLI, explained here:  
* https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/, Journal SSD Recommendations
 
Having that said, initializing an OSD can be done with:
 
# pveceph createosd /dev/sd[X]
 
If you want to use a dedicated SSD journal disk:
 
# pveceph createosd /dev/sd[X] -journal_dev /dev/sd[X]
 
Example: /dev/sdf as data disk (4TB) and /dev/sdb is the dedicated SSD journal disk
 
# pveceph createosd /dev/sdf -journal_dev /dev/sdb
 
This partitions the disk (data and journal partition), creates filesystems, starts the OSD and add it to the existing crush map. So afterward the OSD is running and fully functional. Please create at least 12 OSDs, distributed among your nodes (4 on each node).
 
It should be noted that this command refuses to initialize disk when it detects existing data. So if you want to overwrite a disk you should remove existing data first. You can do that using:
 
# ceph-disk zap /dev/sd[X]
*In some cases disks that used to be part of a 3ware raid need the following in addition to zap.
 
try this:
<pre>
#To remove partition table and boot sector the following should be sufficient:
dd if=/dev/zero of=/dev/$DISK bs=1024 count=1
</pre>
or
<pre>
DISK=$1
 
if [ "$1" = "" ]; then
echo "Need to supply a  dev name like  sdg . exiting"
exit 1
fi
echo " make sure this is the correct disk "
echo $DISK
echo " you will end up with NO partition table when this procedes .  example:
Disk /dev/$1  doesn't contain a valid partition table
Press enter to procede , or ctl-c to exit "
 
read x
dd if=/dev/zero of=/dev/$DISK bs=512 count=50000
</pre>
 
You can create OSDs containing both journal and data partitions or you can place the journal on a dedicated SSD. Using a SSD journal disk is highly recommended if you expect good performance.
 
'''Note:'''
 
In order to use a dedicated journal disk (SSD), the disk needs to have a GPT partition table. You can create this with 'gdisk /dev/sd(x)'. If there is no GPT, you cannot select the disk as journal. Currently the journal size is fixed to 5 GB.
 
== Ceph Pools ==
 
[[Image:Screen-Ceph-Pools.png|thumb]] [[Image:Screen-Ceph-Log.png|thumb]] The standard installation creates some default pools, so you can either use the standard 'rbd' pool, or create your own pools using the GUI.
 
In order to calculate your the  number of placement groups for your pools, you can use:
 
'''Ceph PGs per Pool Calculator'''
 
http://ceph.com/pgcalc/
 
== Ceph Client ==
 
You can then configure Proxmox VE to use such pools to store VM images, just use the GUI ("Add Storage": RBD). A typical entry in the Proxmox VE storage configuration looks like:
 
# from /etc/pve/storage.cfg
rbd: my-ceph-storage
      monhost 10.10.10.1;10.10.10.2;10.10.10.3
      pool rbd
      content images
      username admin
      krbd 0
 
 
You also need to copy the keyring to a predefined location.
:''Note that the file name needs to be storage id + .keyring '' .  '''storage id is the expression after 'rbd:' in /etc/pve/storage.cfg ''' which is ''my-ceph-storage'' in the current example.
 
# cd /etc/pve/priv/
# mkdir ceph
# cp /etc/ceph/ceph.client.admin.keyring ceph/my-ceph-storage.keyring
 
If you want to store containers on Ceph, you need to create an extra pool using KRBD.
 
# from /etc/pve/storage.cfg
rbd: my-ceph-storage-for-lxc
      monhost 10.10.10.1;10.10.10.2;10.10.10.3
      pool rbd-lxc
      content images
      username admin
      krbd 1
 
== Set the Ceph OSD tunables ==
 
ceph osd crush tunables hammer
 
If you set the tunable and your ceph cluster is used you should do this when the cluster has the least load.
Also try to use the backfill option to do this process slower.
 
[http://docs.ceph.com/docs/jewel/dev/osd_internals/backfill_reservation/ Ceph Backfill]
 
'''Important'''
Do not set tunable to '''optimal''' because then krbd will not work.
Set it to hammer.
 
== Why do we need a new command line tool (pveceph)? ==
 
For the use in the specific Proxmox VE architecture we use pveceph. Proxmox VE provides a distributed file system ([http://pve.proxmox.com/wiki/Proxmox_Cluster_file_system_%28pmxcfs%29 pmxcfs]) to store configuration files.
 
We use this to store the Ceph configuration. The advantage is that all nodes see the same file, and there is no need to copy configuration data around using ssh/scp. The tool can also use additional information from your Proxmox VE setup.
 
Tools like ceph-deploy cannot take advantage of that architecture.
 
== Note for users of HP SmartArray controllers ==
 
Proxmox will struggle to add an OSD on a SmartArray conroller via the GUI. This is due to an '!' being inserted into the device path, where a '/' should be instead. An example would be that "/dev/cciss!c0d0" is shown instead of "/dev/cciss/c0d0".
 
This is, however, very easy to fix with a small file edit.
 
Connect to the terminal via SSH.
 
Open the following file in your editor of choice (Vi, Emacs, Nano, etc):
 
'''/usr/share/perl5/PVE/API2/Ceph.pm'''
Find the line that says:
 
'''$devname =~ s|/dev/||;'''
Add the following string on a new line, immediately after the line mentioned above:
 
'''$devname =~ s|cciss/|cciss!|;'''
 
Finally, save the file.
 
Unfortunately, this will still not allow the disk to be added via the GUI, but you can now use the instructions above to add the OSD via the CLI, and then carry on as you otherwise would with creation of pools, etc.
 
== Upgrading existing Ceph Server from Hammer to Jewel ==
See [[Ceph Hammer to Jewel]]


== Video Tutorials ==
== Video Tutorials ==
Line 288: Line 19:
You can subscribe to our [http://www.youtube.com/ProxmoxVE Proxmox VE Channel] on YouTube to get updates about new videos.
You can subscribe to our [http://www.youtube.com/ProxmoxVE Proxmox VE Channel] on YouTube to get updates about new videos.


== Further readings about Ceph ==
== Ceph Misc ==
 
Ceph comes with plenty of documentation [http://ceph.com/docs/master/ here]. Even better, the dissertation from the creator of Ceph - Sage A. Weil - is also [http://ceph.com/papers/weil-thesis.pdf available]. By reading this you can get a deep insight how it works.


*http://ceph.com/
=== Upgrading existing Ceph Server ===
*https://www.redhat.com/en/technologies/storage/ceph


*https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/, Journal SSD Recommendations
* From Hammer to Jewel: See [[Ceph Hammer to Jewel]]


== notes ==
* From Jewel to Luminous: See [[Ceph Jewel to Luminous]]
after ceph is set up and running  make only one change at a time.


adding and osd,  changing pool settings -  check log and make sure health is normal before the next change.  too many changes at the same time can result in  slow systems - bad for cli.
for instance,  note ceph -s from cli  for these:  [    pve  ceph >logs  also shows activity ]
* osd added  I/O changes in progress:
<pre>
sys4  /etc/network # ceph -s
    cluster 63efaa45-7507-428f-9443-82a0a546b70d
    health HEALTH_WARN
            14 pgs backfill_wait
            1 pgs backfilling
            15 pgs stuck unclean
            recovery 3893/303456 objects misplaced (1.283%)
    monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
            election epoch 170, quorum 0,1,2 1,0,2
    osdmap e1122: 10 osds: 10 up, 10 in; 15 remapped pgs
            flags sortbitwise,require_jewel_osds
      pgmap v1789114: 800 pgs, 2 pools, 391 GB data, 100488 objects
            1174 GB used, 3244 GB / 4419 GB avail
            3893/303456 objects misplaced (1.283%)
                785 active+clean
                  14 active+remapped+wait_backfill
                  1 active+remapped+backfilling
recovery io 358 MB/s, 90 objects/s
  client io 45235 B/s rd, 1116 kB/s wr, 1 op/s rd, 161 op/s wr
</pre>
*done , and ready for next change:
<pre>
sys4  /etc/network # ceph -s
    cluster 63efaa45-7507-428f-9443-82a0a546b70d
    health HEALTH_OK
    monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
            election epoch 170, quorum 0,1,2 1,0,2
    osdmap e1146: 10 osds: 10 up, 10 in
            flags sortbitwise,require_jewel_osds
      pgmap v1789240: 800 pgs, 2 pools, 391 GB data, 100488 objects
            1174 GB used, 3244 GB / 4419 GB avail
                800 active+clean
  client io 1516 kB/s rd, 300 kB/s wr, 15 op/s rd, 57 op/s wr
</pre>
*or
ceph -s|grep HEALTH
===using a disk that was part of a zfs pool ===
as of now
  ceph-disk zap /dev/sd?
is needed.
else it does not show up on pve > ceph > osd > Create OSD
=== restore lxc from zfs to ceph ===
=== restore lxc from zfs to ceph ===
if lxc is on zfs with compression  the actual disk usage can be far greater then expected.  
if lxc is on zfs with compression  the actual disk usage can be far greater then expected.  
Line 357: Line 35:
=== scsi setting ===
=== scsi setting ===
make sure that you use  virtio-scsi controller (not LSI),  see VM options. I remember some panic when using LSI recently but I did not debug it further as modern OS should use virtio-scsi anyways. https://forum.proxmox.com/threads/restarted-a-node-some-kvms-on-other-nodes-panic.32806
make sure that you use  virtio-scsi controller (not LSI),  see VM options. I remember some panic when using LSI recently but I did not debug it further as modern OS should use virtio-scsi anyways. https://forum.proxmox.com/threads/restarted-a-node-some-kvms-on-other-nodes-panic.32806
=== noout ===
http://docs.ceph.com/docs/jewel/rados/troubleshooting/troubleshooting-osd/#stopping-w-out-rebalancing
"Periodically, you may need to perform maintenance on a subset of your cluster, or resolve a problem that affects a failure domain (e.g., a rack). If you do not want CRUSH to automatically rebalance the cluster as you stop OSDs for maintenance, set the cluster to noout first:"
ceph osd set noout
'''that is so important that it can be done from pve at  ceph>osd'''


[[Category:HOWTO]] [[Category:Installation]] [[Category:Technology]]
[[Category:HOWTO]] [[Category:Installation]]

Latest revision as of 14:05, 12 January 2024

This article got replaced by our reference documentation's ceph chapter: https://pve.proxmox.com/pve-docs/chapter-pveceph.html

Recommended hardware

Take a look at our Proxmox VE Ceph Benchmark 2018/02 for possible hardware decisions. https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark

Further readings about Ceph

Ceph comes with plenty of documentation here. Even better, the dissertation from the creator of Ceph - Sage A. Weil - is also available. By reading that thesis you can get an in-depth insight how Ceph works.

Video Tutorials

Proxmox YouTube channel

You can subscribe to our Proxmox VE Channel on YouTube to get updates about new videos.

Ceph Misc

Upgrading existing Ceph Server

restore lxc from zfs to ceph

if lxc is on zfs with compression the actual disk usage can be far greater then expected. see https://forum.proxmox.com/threads/lxc-restore-fail-to-ceph.32419/#post-161287

One way to know actual disk usage:

restore backup to a ext4 directory and run du -sh , then do restore manually specifying target disk size.

scsi setting

make sure that you use virtio-scsi controller (not LSI), see VM options. I remember some panic when using LSI recently but I did not debug it further as modern OS should use virtio-scsi anyways. https://forum.proxmox.com/threads/restarted-a-node-some-kvms-on-other-nodes-panic.32806

Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Ceph_Server&oldid=11909"