HTTPS Certificate Configuration (Version 4.x, 5.0 and 5.1)

Introduction

This is a howto for changing the web server certificate used by Proxmox VE, in order to enable the usage of publicly trusted certificates issued by a CA of your choice (like Let's Encrypt or a commercial CA). It has been tested on a Proxmox VE 4.1 installation, using certificates from https://www.letsencrypt.org.

Note: the previous, outdated version of this HowTo is archived at HTTPS Certificate Configuration (Version 3.x and earlier)

Revert to default configuration

If you have used the previous HowTo and replaced any of the certificate or key files generated by PVE, you need to revert to the default state before proceeding.

Delete or move the following files:

• /etc/pve/pve-root-ca.pem
• /etc/pve/priv/pve-root-ca.key
• /etc/pve/nodes/<node>/pve-ssl.pem
• /etc/pve/nodes/<node>/pve-ssl.key

The latter two need to be repeated for all nodes if you have a cluster.

Afterwards, run the following command on each node of the cluster to re-generate the certificates and keys:

pvecm updatecerts -f

CAs other than Let's Encrypt

Install certificate chain and key

Since pve-manager 4.1-20, it is possible to provide alternative SSL files for each node's web interface. The following steps need to be repeated for each node where you want to use alternative certificate files.

First check your version of pve-manager and upgrade if necessary:

pveversion

You will need the following two files provided by your CA:

• fullchain.pem (your certificate and all intermediate certificates, excluding the root certificate, in PEM format)
• private-key.pem (your private key, in PEM format, without a password)

Now copy those files to the override locations in /etc/pve/nodes/<node> (make sure to use the correct certificate files and node!):

cp fullchain.pem /etc/pve/nodes/<node>/pveproxy-ssl.pem
cp private-key.pem /etc/pve/nodes/<node>/pveproxy-ssl.key

and restart the web interface:

systemctl restart pveproxy

The system log should inform you about the usage of the alternative SSL certificate ("Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface."):

journalctl -b -u pveproxy.service

When accessing the web interface on this node, you should be presented with the new certificate. Note that the alternative certificate is only used by the web interface (including noVNC), but not by the Spice Console/Shell.

Let's Encrypt using acme.sh

Prerequisites

Let's Encrypt enables everyone with a publicly resolvable domain name to be issued SSL certificates for free.

Your domain name needs to be publicly resolvable both ways. (Check with `$ drill -x Your.Ip.Address` or `$ dig -x Your.Ip.Address`)

The following steps show how to achieve this using the acme.sh bash script and standalone HTTP authentication.

These steps need to be repeated on each node where you want to use Let's Encrypt certificates.

You need at least pve-manager >= 4.1-20 (see `$ pveversion`), so upgrade if necesasry.

Install certificate chain and key

0) Upgrade from le.sh to acme.sh

If you followed a previous version of this HowTo using le.sh, please uninstall le.sh and proceed with "Install acme.sh":

le.sh uninstall

acme.sh is the 2.X release of le.sh, the existing configuration should be migrated automatically when installing acme.sh.

1) Install acme.sh

Install the acme.sh script from https://github.com/Neilpang/acme.sh (this howto was tested with commit 2d39b3df8893cd256257fe1f32ca6b0485a90dcf):

Via git:

git clone https://github.com/Neilpang/acme.sh.git acme.sh-master

Or direct download:

wget 'https://github.com/Neilpang/acme.sh/archive/master.zip'
unzip master.zip

2) Run the install script

You must do this from within the script's directory, otherwise it won't find acme.sh! Take care to replace $EMAIL with the address that you want to register with at Let's Encrypt. Let's Encrypt will send automatic expiration reminders to this address!

mkdir /etc/pve/.le
cd /root/acme.sh-master
./acme.sh --install --accountconf /etc/pve/.le/account.conf --accountkey /etc/pve/.le/account.key --accountemail "$EMAIL"

After the installation, you need to close and reopen your shell (acme.sh is installed to ~/.acme.sh and sets up your shell's $PATH accordingly).

3) Check the account config

Check the config file in /etc/pve/.le/account.conf and verify:

• the ACCOUNT_EMAIL variable should be set to your email address
• the ACCOUNT_KEY_PATH variable should be set to "/etc/pve/.le/account.key"

You can edit this file with your favourite text editor if either of those is incorrect.

4) Make sure port 80 is open from the public

As part of the certificate creation process, acme.sh will listen for a confirmation from LetsEncrypt's servers on port 80. Check that this port is therefore not blocked by any firewall between the machine you are certifying and the public internet.

You can close the port once you're done issuing all certificates for your cluster. However, be aware that as part of the certificate renewal process (managed with a cron job that acme.sh installs), port 80 must also be open. You may therefore need to work out an automated way (not covered in this guide) of opening up port 80 for the renewal process.

5) Issue your first certificate

Now you can issue your first certificate, replacing $DOMAIN with your node's fully qualified domain:

acme.sh --issue --standalone --keypath /etc/pve/local/pveproxy-ssl.key --fullchainpath /etc/pve/local/pveproxy-ssl.pem --reloadcmd "systemctl restart pveproxy" -d $DOMAIN

Warnings like "cp: preserving permissions for ‘/etc/pve/local/pveproxy-ssl.pem.bak’: Function not implemented" can be safely ignored.

By appending the previous command with --test you can issue a certificate using the staging (i.e., testing) CA instead of the production CA:

acme.sh --issue --standalone --keypath /etc/pve/local/pveproxy-ssl.key --fullchainpath /etc/pve/local/pveproxy-ssl.pem --reloadcmd "systemctl restart pveproxy" -d $DOMAIN --test

To "upgrade" to a production certificate, you need to rerun the issue command with an appended --force instead of --test, in order to replace the existing (test) certificate even though it is not yet expired. This can also be used to force a premature renewal in case the node's domain name has changed:

acme.sh --issue --standalone --keypath /etc/pve/local/pveproxy-ssl.key --fullchainpath /etc/pve/local/pveproxy-ssl.pem --reloadcmd "systemctl restart pveproxy" -d $DOMAIN --force

6. Check it's working

If necessary, close the firewall port again.

The system log should inform you about the usage of the alternative SSL certificate ("Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface."):

journalctl -b -u pveproxy.service

When accessing the web interface on this node, you should be presented with the new certificate. Note that the alternative certificate is only used by the web interface (including noVNC), but not by the Spice Console/Shell.

7. Set up automatic renewal

acme.sh installs a cron job that checks the installed certificate(s) and automatically renews them before they expire.

The crontab entry should look like this (crontab -l):

0 0 * * * "/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh" > /dev/null

It's a good idea to test the cron entry by running it manually from the command line, just copy the following command (including the quotes):

"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

NOTE: The requirements for issuing certificates apply for renewals as well: the configured domain name must be resolvable and reachable on port 80 from the public internet when the renewal cron job runs.

Updating acme.sh

acme.sh can be updated with the following commands when installed from the git repository:

cd /root/acme.sh-master
git pull
./acme.sh --install --accountconf /etc/pve/.le/account.conf --accountkey /etc/pve/.le/account.key --accountemail "YOUR@EMAIL.ADDRESS"

Account key

It is recommended to do an off-site/offline backup of the account key file in /etc/pve/.le/account.key, in case one of your certificate private key files is lost or compromised, it can be used to revoke the associated certificate.

Let's Encrypt using eff.org's certbot

Installing certbot

certbot is packaged in Debian (the building base of Proxmox VE), so you may just install it with apt:

apt install certbot

If you run PVE 4.X (Debian Jessie) or run into any problems which are fixed by a newer version, you may enable the backports repository ( http://backports.debian.org/Instructions/ ) and run:

apt -t stretch-backports install certbot

or respectively:

apt -t jessie-backports install certbot

Obtaining your certificate

Just run the following command and follow the on screen output to obtain your certificate.

certbot certonly

Getting your certificate into Proxmox

You need to copy the certificate from the LE directory to the PVE directory:

cp /etc/letsencrypt/live/<domain>/fullchain.pem /etc/pve/local/pveproxy-ssl.pem
cp /etc/letsencrypt/live/<domain>/privkey.pem /etc/pve/local/pveproxy-ssl.key

Than restart the PVE proxy:

systemctl restart pveproxy

Setting up renewal

Create a new bash script with the commands used to copy the certificate and restart the proxy from the previous step and make it executable, lets assume that you put it into /usr/local/bin/renew-pve-certs.sh

Than edit your /etc/crontab with the editor your like and add the following line:

30 6 1,15 * * root /usr/bin/certbot renew --quiet --post-hook /usr/local/bin/renew-pve-certs.sh

This will check two times each month (on the first and the 15th) at 06:30 AM if it needs to issue a new certificate. As certificates are valid for three months, this frequency should be high enough to ensure a certificate gets always renewed on time.

Let's Encrypt using other Clients

It should also be possible to use other Let's Encrypt clients, as long as care is taken that the issued as well as renewed certificates and the associated keys are copied to the correct locations, and the pveproxy service is restarted afterwards.