Booting a ZFS root file system via UEFI

If you need to boot a ZFS root filesystem you already have on a new system which only supports UEFI, you cannot do this out-of-the-box. Unfortunately, there is current no capability to do this in ZFS on Linux, so Proxmox VE is unfortunately also not able to do this (Also mentioned in the article about ZFS on Linux). However, you can do this by using a UEFI boot stick with rEFInd, and manually sync it.

You can use whatever medium you like:

• another internal (bootable) disk
• USB flash drive
• network
• optical disk

for simplicity, this article is about an USB flash drive.

WARNING

This is not recommended, especially not in a production environment, where you create a single point of failure. Please be aware that you should only consider using this setup if you know what you're doing and are familiar with the Linux command line, its system tools and program, and do it on your own risk. If you use the wrong device, you could nuke your whole system, so consider yourself warned.

Software Introduction

The rEFInd Boot Manager by Roderick W. Smith is an UEFI boot manager, which is able to boot a lot of different operating systems including Linux and therefore Proxmox VE. Its setup and configuration is very easy.

Preparting for rEFInd

Not all programs used in this article are already preinstalled in your Proxmox VE environment, so please install them first:

apt install lsscsi wget fdisk unzip parted

Using an USB flash drive

Double check, that you're using the correct drive, this article will assume the drive is available in /dev/sdi. The minimum requirement for a thumb drive is 128 MB (note the M, so any drive from the last 10 years should be sufficient).

root@proxmox ~ > lsscsi --size | grep sdi
[7:0:0:0]    disk    JetFlash Transcend 32GB   1.00  /dev/sdi   31.6GB

Let's create the partition layout (for simplicity, only one partition)

root@proxmox ~ > parted -s /dev/sdi -- mktable gpt
root@proxmox ~ > parted -s /dev/sdi -- mkpart UEFI fat32 1 -1
root@proxmox ~ > parted -s /dev/sdi -- print
Model: JetFlash Transcend 32GB (scsi)
Disk /dev/sdi: 31,6GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  31,6GB  31,6GB               UEFI  msftdata

and filesystem

root@proxmox ~ > mkfs.vfat -F32 /dev/sdi1
mkfs.fat 4.1 (2017-01-24)

The setup is similar to an internal disk setup, but I'd create a smaller partition.

Setting up rEFInd for Proxmox VE

Installing rEFInd

This article describes the manual install method according to the official documentation.

After mounting our newly created UEFI boot disk from the previous step:

root@proxmox ~ > mount /dev/sdi1  /mnt

we download the binary distribution from the Getting rEFInd page directly onto our flash drive (please always get the download link from the aforementioned webpage to use the most recent version of rEFInd):

root@proxmox /mnt > wget -q -O refind-bin-0.11.3.zip http://sourceforge.net/projects/refind/files/0.11.3/refind-bin-0.11.3.zip/download
root@proxmox /mnt > ls -l refind-bin-0.11.3.zip
-rwxr-xr-x 1 root root 3702691 Jul 22 23:43 refind-bin-0.11.3.zip

and extract it

root@proxmox /mnt > unzip -q refind-bin-0.11.3.zip
root@proxmox /mnt > ls -l
total 3632
drwxr-xr-x 7 root root   16384 Jul 22 23:08 refind-bin-0.11.3
-rwxr-xr-x 1 root root 3702691 Jul 22 23:43 refind-bin-0.11.3.zip

now install the binary distribution of rEFInd - which is a simple copy job

root@proxmox /mnt > mkdir -p efi/boot
root@proxmox /mnt > cp -r refind-bin-0.11.3/refind/{drivers_x64,icons,tools_x64} efi/boot/
root@proxmox /mnt > cp refind-bin-0.11.3/refind/refind_x64.efi efi/boot/bootx64.efi

Afterwards, you already have a working UEFI bootloader. It uses its defaults settings and scans for all available boot options and present them graphically. You can remove the install files, but it's not that big and you can keep it for further reference.

Configuring rEFInd for Proxmox VE

In this simple example, it boots only the preconfigured Proxmox VE kernel and the current system. In a more advanced setup, you can do whatever you want - but it is out of the scope of this article.

First, we copy the current (at the time of writing 4.15.18-5-pve) kernel to the efi device and list the important files:

root@proxmox /mnt > cp /boot/{initrd.img,vmlinuz}-4.15.18-5-pve .

root@proxmox /mnt > l
total 59296
drwxr-xr-x 3 root root    16384 Sep 30 09:28 efi
-rwxr-xr-x 1 root root 48528392 Sep 30 09:45 initrd.img-4.15.18-5-pve
-rwxr-xr-x 1 root root  8443760 Sep 30 09:45 vmlinuz-4.15.18-5-pve

Now create a configuration file in efi/boot/refind.conf with the following content:

timeout 20
icons_dir EFI/boot/icons/
scanfor manual
scan_all_linux_kernels false

menuentry "Proxmox VE" {
    ostype Linux
    graphics on
    loader /vmlinuz-4.15.18-5-pve
    initrd /initrd.img-4.15.18-5-pve
    options "ro root=ZFS=rpool/ROOT/pve-1 quiet nomodeset"
}

Unmount the pen drive:

root@proxmox ~ > umount /mnt

You should now have a working UEFI boot stick which boots directly to your default Proxmox VE ZFS rpool.

Eye Candy

If you also want to use a nice icon for your private setup, just search on google image search for a nice looking Proxmox VE logo or create one yourself and put it with transparent background in a PNG file. A good starting point is also the PVE Wiki logo.

You can should put the file into your root directory and the settings like this:

root@proxmox /mnt > l
total 59312
drwxr-xr-x 3 root root    16384 Sep 30 09:28 efi
-rwxr-xr-x 1 root root 48528392 Sep 30 09:45 initrd.img-4.15.18-5-pve
-rwxr-xr-x 1 root root     5382 Sep 30 10:11 proxmox.png
-rwxr-xr-x 1 root root  8443760 Sep 30 09:45 vmlinuz-4.15.18-5-pve

root@proxmox /mnt > cat efi/boot/refind.conf
timeout 20
icons_dir EFI/boot/icons/
scanfor manual
scan_all_linux_kernels false

menuentry "Proxmox VE" {
    icon /proxmox.png
    ostype Linux
    graphics on
    loader /vmlinuz-4.15.18-5-pve
    initrd /initrd.img-4.15.18-5-pve
    options "ro root=ZFS=rpool/ROOT/pve-1 quiet nomodeset"
}

Afterwards, you end up with this screen (the little pen drive logo on the right bottom implies, that this entry will be booted from USB):

Update

Unfortunately, you have to update your settings, kernel and initrd manually in order to boot newer updates, but as it is said in the beginning, this is totally unofficial and not for production use.

Normally, these steps are required:

• coping vmlinuz and initrd.img files corresponding to the current version to your UEFI disk
• adding a new section or changing the current one (better to add a new entry in order to be able to revert to the old kernel in case of error or misconfiguration)
• reboot