Automated Installation

This article documents the automated installation process of Proxmox projects like Proxmox VE.

Introduction

The automated installation method allows installing a Proxmox solution in an unattended manner. This enables you to fully automate the setup process on bare-metal. Once the installation is complete and the host has booted up, automation tools like Ansible can be used to further configure the installation.

The necessary options for the installer must be provided in an answer file. This file allows using filter rules to determine which disks and network cards should be used.

In order to use the automated installation it is first necessary to choose a source from which the answer file is fetched from and then prepare an installation ISO with that choice.

Once the ISO is prepared, its initial boot menu will show a new boot entry named "Automated Installation" which gets automatically selected after a 10-second timeout.

Overview

Assistant Tool

The proxmox-auto-install-assistant tool provides the prepare-iso sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation. You will have to install it first:

apt install proxmox-auto-install-assistant

NOTE: The xorriso binary is required for preparing an ISO. On Debian-based systems you can install it using apt install xorriso.

See the help of the sub-command has more details: proxmox-auto-install-assistant prepare-iso --help.

Answer File Format

The answer file is a TOML-formatted configuration file that provides the basic configuration for a system, like the root password, the network configuration and the target root disk.

In order to allow installation of systems with (for example) many disks and network devices you can use filters to match on device properties, such as serial numbers, vendor or other unique details like the MAC address of a network interface.

For more details about the schema and the possible filters see the Answer File Format section.

Answer File Source

There are several locations where the automatic installer can fetch an answer file from:

• from the ISO directly
• read from a separate partition with the PROXMOX-AIS partition label
• fetched through HTTP from the network
  • the source URL can be pre-determined or queried from DNS or DHCP

For now, you must choose exactly one source.

Prepare an Installation ISO

See the following for a more detailed description of the possible sources and examples on how to prepare a ISO with the different fetch-from modes.

Answer included in the ISO

It is possible to prepare an ISO so that it includes an answer.toml directly, allowing you to have a unified medium for rolling out automatic installation.

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from iso --answer-file /path/to/answer.toml

Answer on Separate Partition

The ISO can be prepared on a file system labelled proxmox-ais or PROXMOX-AIS (Automated Installation Source) containing the answer.toml file (for example, on a USB flash drive).

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from partition

Afterwards, prepare the USB flash drive, for example on /dev/sdX1. You may then adapt and run the following commands as root:

# format as new vfat file system (DESTRUCTIVE!)
mkfs.vfat /dev/sdX1

# set the label
fatlabel /dev/sdX1 "PROXMOX-AIS"

# mount the USB pen drive 
mkdir /mnt/usb
mount /dev/sdX1 /mnt/usb

# copy over the answer file as "answer.toml"
cp my-prepared-answer.toml /mnt/usb/answer.toml

# unmount the USB pen drive
sync
umount /mnt/usb

Answer Fetched via HTTP

In order to fetch the answer file via HTTP(S), the installer needs to know the URL. The URL may be provided via the following ways:

• URL defined in the ISO
• as DHCP option (250)
• via a DNS TXT record
  • The DNS TXT record needs to be located at proxmox-auto-installer.{search domain}, where {search domain} is the search domain provided by the DHCP server.

Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

The HTTP(S) POST request sends JSON data that can help to identify the physical machine and use that information to generate a custom answer file.

You can see how this information looks like by using the sudo proxmox-auto-install-assistant system-info command.

Certificate Fingerprint Matching

It is possible to provide the 'SHA256' certificate fingerprint of the TLS certificate. This is useful in the following situations:

• the URL is using an IP address instead of a FQDN
• a self-signed certificate is used that the installer does not trust
• added security by pinning the known certificate of the server

There are three ways to provide the 'SHA256' fingerprint to the installer:

• Fingerprint defined in the ISO
• as a DHCP option (251)
• via a DNS TXT record
  • The DNS TXT record must be located at proxmox-auto-installer-cert-fingerprint.{search domain}, where {search domain} is the search domain provided by the DHCP server.

Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

Example

Similarly, it is possible to specify the URL from which the installer should fetch an answer file as well as the TLS certificate fingerprint. For example, to specify both:

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from http --url "https://10.0.0.100/get_answer/" --cert-fingerprint "04:42:97:27:F6:29:2F:9F:3D:7F:13:11:C8:E2:F5:5F:84:03:95:D9:F5:14:72:7C:9E:90:47:03:D2:96:2B:EC"

Answer File Format

The answer file is expected in TOML format.

The following example shows a basic answer file that uses the DHCP-provided network settings and will use ZFS in a RAID-1 on disks sda and sdb:

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
root_ssh_keys = [
    "ssh-ed25519 AAAA..."
]

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
disk_list = ["sda", "sdb"]

Global Section

This section contains the following keys:

• keyboard -- The keyboard layout with the following possible options:

 de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr 

• country -- The country code in the two letter variant. For example, at, us or fr.
• fqdn -- The fully qualified domain name of the host. The domain part will be used as the search domain.
• mailto -- The default email address for the user root.
• timezone -- The timezone in tzdata format. For example, Europe/Vienna or America/New_York.
• root_password -- The password for the root user.
• root_ssh_keys -- Optional. SSH public keys to add to the root users authorized_keys file after the installation.
• reboot_on_error -- If set to true, the installer will reboot automatically when an error is encountered. The default behavior is to wait to give the administrator a chance to investigate why the installation failed.

Network Section

This section contains the following keys:

• source -- Where to source the static network configuration from. This can be from-dhcp or from-answer.

  If set to from-dhcp the other network options will be ignored and the installer will use the active NIC and the DHCP settings received during installation to write out a static network configuration.

• cidr -- The IP address in CIDR notation. For example, 192.168.1.10/24
• dns -- The IP address of the DNS server.
• gateway -- The IP address of the default gateway.
• filter -- Filter against the UDEV properties to select the network card. See filters.

Disk Setup Section

This section contains the following keys:

• filesystem -- One of the following options: ext4, xfs, zfs, or btrfs.
• disk_list -- List of disks to use. Useful if you are sure about the disk names. For example: disk_list = ["sda", "sdb"]
• filter -- Filter against UDEV properties to select the disks for the installation. See filters.

  NOTE: Use either disk_list or filter. Defining both is not allowed.

• filter_match -- Can be "any" or "all". Decides if a match of any filter is enough of if all filters need to match for a disk to be selected. Default is "any".
• zfs -- Defines ZFS-specific properties. See ZFS Advanced Options of our documentation. The properties are:
  •  raid -- The RAID level that should be used. Options are raid0, raid1, raid10, raidz-1, raidz-2, or raidz-3.
  •  ashift
  •  arc_max
  •  checksum
  •  compress
  •  copies
  •  hdsize
• lvm -- Advanced properties that can be used with the ext4 or xfs file system. See LVM Advanced Options. The properties are:
  •  hdsize
  •  swapsize
  •  maxroot
  •  maxvz
  •  minfree
• btrfs -- Defines BTRFS specific options.
  •  raid -- The RAID level that should be used. Options are raid0, raid1, and raid10.
  •  hdsize

Answer File Validation

The proxmox-auto-install-assistant tool can also be used to validate the syntax of an answer file and display the identifying information that will be sent to the HTTP(s) server when fetching the answer file.

For example, to validate an answer file:

$ proxmox-auto-install-assistant validate-answer answer.toml 
The file was parsed successfully, no syntax errors found!

Filters

Filters allow you to match against device properties exposed by udevadm.

The proxmox-auto-install-assistant utility can display these properties and allows you to test filters in advance. The utility is available in the installer environment (via its debug mode) and on already existing Proxmox Virtual Environment installation.

For example, to fetch information about the available disks:

$ proxmox-auto-install-assistant device-info -t disk
[…]
    "nvme1n1": {
      "CURRENT_TAGS": ":systemd:",
      "DEVLINKS": "/dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1 /dev/disk/by-path/pci-0000:e2:00.0-nvme-1 /dev/disk/by-diskseq/12 /dev/disk/by-id/nvme-eui.01000000000000008ce38ee300708529 /dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX",
      "DEVNAME": "/dev/nvme1n1",
      "DEVPATH": "/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1",
      "DEVTYPE": "disk",
      "DISKSEQ": "12",
      "ID_MODEL": "KIOXIA KCMYXVUG1T60",
      "ID_NSID": "1",
      "ID_PART_TABLE_TYPE": "gpt",
      "ID_PART_TABLE_UUID": "539fabeb-aecd-6643-94a9-f28a68cfa12d",
      "ID_PATH": "pci-0000:e2:00.0-nvme-1",
      "ID_PATH_TAG": "pci-0000_e2_00_0-nvme-1",
      "ID_REVISION": "1UETE103",
      "ID_SERIAL": "KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1",
      "ID_SERIAL_SHORT": "9DU0A02D0L33",
      "ID_WWN": "eui.01000000000000008ce38ee300708529",
      "MAJOR": "259",
      "MINOR": "5",
      "SUBSYSTEM": "block",
      "TAGS": ":systemd:",
      "USEC_INITIALIZED": "3169050"
    },
[…]

The key of the filter decides on which property it should be applied to. For example, in order to match against the vendor and model number of the disk, the filter in the answer file could look like this:

filter.ID_SERIAL = "KIOXIA_KCMYXVUG1T60*"

Note: The * globbing symbol at the end is used to match anything after the defined filter!

This will match for all Kioxia disks with that model number. You may verify which disks will be found by the filter by running the following command:

$ proxmox-auto-install-assistant device-match disk ID_SERIAL='KIOXIA_KCMYXVUG1T60*'
[
  "nvme0n1",
  "nvme1n1",
  "nvme4n1",
  "nvme5n1"
]

The filter_match parameter controls whether all filters must apply, or if it is enough if any of the filters match. This makes it possible to use different disk models for the installation by using different properties.

For example:

filter.ID_SERIAL = "KIOXIA*"
filter.ID_MODEL = "ATP*"

Note: For network cards, only the first match will be used as the installer requires only one network card.

More complex network setups can be configured after the installation. Using properties with unique identifiers will result in the most predictable behavior (for example, the MAC address).

Filter Syntax

The following special characters can be used in filters:

• ? -- matches any single characters
• * -- matches any number of characters, can be none
• [a], [abc], [0-9] -- matches any single character inside the brackets, ranges are possible
• [!a] -- negate the filter, any single character but the ones specified

Useful Properties

For network cards, the following properties can be useful:

• ID_NET_NAME
• ID_NET_NAME_MAC
• ID_VENDOR_FROM_DATABASE
• ID_MODEL_FROM_DATABASE

For disks, these properties can be useful:

• DEVNAME
• ID_SERIAL_SHORT
• ID_WWNKIOXIA_KCMYXVUG1T60
• ID_MODEL
• ID_SERIAL

Examples

Answer Files

ZFS Mirror on Samsung Disks

We assume that there are only two Samsung disks present. These should be used for the OS in a ZFS Mirror (RAID-1). Additionally, we want to use only 150 GiB of the disks' capacities and leave the rest empty for further customization.

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
zfs.hdsize = 150
filter.ID_MODEL = "Samsung*"

Ext4 With No Swap and Data LV on /dev/sda

Install on /dev/sda and define swap and data LVs with size 0.

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "ext4"
lvm.swapsize = 0
lvm.maxvz = 0
disk_list = ['sda']

ZFS Mirror With Manual Network Config

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-answer"
cidr = "10.10.10.10/24"
dns = "10.10.10.1"
gateway = "10.10.10.1"
filter.ID_NET_NAME_MAC = "*e43d1afa379a"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
filter.ID_MODEL = "Samsung*"

Note: There is a * in the filter for the network card. This is necessary, because that property usually has enx prefixed.

When displaying this property with proxmox-auto-install-assistant device-info -t network, its full value looks like this:

"ID_NET_NAME_MAC": "enxe43d1afa379a",

Troubleshooting

If the installation fails for some reason, it will drop into a shell (unless the reboot_on_error option in the answer file is set to true). This gives you the chance to troubleshoot what went wrong.

The log files of interest will most likely be:

• /tmp/fetch_answer.log - the steps to retrieve an answer file
• /tmp/auto_installer - parsing of the answer file, matching of hardware to use
• /tmp/install-low-level-start-session.log - the actual installation process