User:Grin/Ceph Object Gateway

Quote from https://www.dragontek.com/blog/post/adding-s3-capabilities-proxmox.

I mostly followed the instructions from the main Ceph site, but it was somewhat confusing because they referred to installing Apache and FCGI in some places, but in others they mention that Ceph uses "Civetweb". There is also mention of using ceph-deploy, but I knew that Proxmox uses it's own pveceph tools. So, not wanting to affect my main Proxmox nodes too much, I decided on my first cut to install a dual NIC VM and put one on the same VLAN as my storage network, and the other on the PVE VLAN. It went well enough, and only required one additional package, so I decided to go ahead and install directly on the Proxmox nodes.

My Proxmox environment consists of 3 nodes: pve1, pve2, and pve3, and I wanted to run the Gateway on all three nodes for High Availabilty (I'm running HAProxy in front of these for SSL termination, HA and load balancing).

I ran the following commands from the pve1 node, but it could have been done from any of the nodes.

First I created the keyring to store the keys:

root@pve1:~# ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring

Next, I generated the keys and added them to the keyring:

root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve1 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve2 --gen-key
root@pve1:~# ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.pve3 --gen-key

And then I added the proper capabilities:

root@pve1:~# ceph-authtool -n client.radosgw.pve1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve2 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph-authtool -n client.radosgw.pve3 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

Finally, I add the keys to the cluster:

root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve1 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve2 -i /etc/ceph/ceph.client.radosgw.keyring
root@pve1:~# ceph -k /etc/ceph/ceph.client.admin.keyring auth add client.radosgw.pve3 -i /etc/ceph/ceph.client.radosgw.keyring

I also copied the keyring into the Proxmox ClusterFS so that it'd be available on all nodes. Note: I might have been able to generate the key directly in the /etc/pve/priv folder, and saved this step.

root@pve1:~# cp /etc/ceph/ceph.client.radosgw.keyring /etc/pve/priv

Add the following lines to /etc/ceph/ceph.conf:

[client.radosgw.pve1]
        host = pve1
        keyring = /etc/pve/priv/ceph.client.radosgw.keyring
        log file = /var/log/ceph/client.radosgw.$host.log
        rgw_dns_name = s3.example.net

[client.radosgw.pve2]
        host = pve2
        keyring = /etc/pve/priv/ceph.client.radosgw.keyring
        log file = /var/log/ceph/client.radosgw.$host.log
        rgw_dns_name = s3.example.net

[client.radosgw.pve3]
        host = pve3
        keyring = /etc/pve/priv/ceph.client.radosgw.keyring
        log file = /var/log/ceph/client.rados.$host.log
        rgw_dns_name = s3.example.net

Here again, I think there's room for optimization. It's my understanding that multiple [client] sections can be combined, so everything below the host line could potentially be merged into a single section to eliminate repetition.

At this point it was time to log into each of the nodes and add the proper packages:

root@pve1:~# apt install radosgw

Create systemd service symlink on each node

root@pve1:~# mkdir /etc/systemd/system/ceph-radosgw.target.wants
root@pve1:~# ln -s /lib/systemd/system/ceph-radosgw@.service /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@radosgw.radosgw.pve1
root@pve1:~# systemctl daemon-reload
root@pve2:~# mkdir /etc/systemd/system/ceph-radosgw.target.wants
root@pve2:~# ln -s /lib/systemd/system/ceph-radosgw@.service /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@radosgw.radosgw.pve2
root@pve2:~# systemctl daemon-reload
root@pve3:~# mkdir /etc/systemd/system/ceph-radosgw.target.wants
root@pve3:~# ln -s /lib/systemd/system/ceph-radosgw@.service /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@radosgw.radosgw.pve3
root@pve3:~# systemctl daemon-reload

And then fire up the gateway per node:

root@pve1:~# systemctl start ceph-radosgw@radosgw.pve1
root@pve2:~# systemctl start ceph-radosgw@radosgw.pve2
root@pve3:~# systemctl start ceph-radosgw@radosgw.pve3

If all goes well, RADOSGW will create some default pools for you (see below), and you should be able to visit any of your nodes on port 7480 (e.g. http://pve1.example.net:7480) and you should see something like this:

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Owner>
        <ID>anonymous</ID>
        <DisplayName/>
    </Owner>
    <Buckets/>
</ListAllMyBucketsResult>

If not, you can follow your logs to troubleshoot:

root@pve1:~# tail -f /var/log/ceph/client.rados.pve1.log

I was getting warnings on my Ceph cluster that the application hadn't been enabled on pools, so I ran the following:

root@pve1:~# ceph osd pool application enable .rgw.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.control rgw
root@pve1:~# ceph osd pool application enable default.rgw.data.root rgw
root@pve1:~# ceph osd pool application enable default.rgw.gc rgw
root@pve1:~# ceph osd pool application enable default.rgw.log rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.uid rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.email rgw
root@pve1:~# ceph osd pool application enable default.rgw.users.keys rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.index rgw
root@pve1:~# ceph osd pool application enable default.rgw.buckets.data rgw
root@pve1:~# ceph osd pool application enable default.rgw.lc rgw

Note: some of these pools showed up only when I needed them, such as creating a user, so I may need to go back and rerun this command with any newly created pools

So now you can setup your first user:

root@pve1:~# radosgw-admin user create --uid=testuser --display-name="Test User" --email=test.user@example.net

That's it for configuration on the servers. If you plan to expose these as I did through HAProxy, don't forget to add a wildcard entry for your domain in DNS: *.s3.example.net, so that your buckets will resolve. I also ended up purchasing a wildcard SSL certificate that I loaded onto HAProxy for SSL.