Automated Installation

From Proxmox VE
Revision as of 13:20, 3 May 2024 by A.lauterer (talk | contribs) (add example server and third party tools sections - now carefully merged)
Jump to navigation Jump to search

This article documents the automated installation process of Proxmox projects like Proxmox VE.

Introduction

The automated installation method allows installing a Proxmox solution in an unattended manner. This enables you to fully automate the setup process on bare-metal. Once the installation is complete and the host has booted up, automation tools like Ansible can be used to further configure the installation.

The necessary options for the installer must be provided in an answer file. This file allows using filter rules to determine which disks and network cards should be used.

In order to use the automated installation it is first necessary to choose a source from which the answer file is fetched from and then prepare an installation ISO with that choice.

Once the ISO is prepared, its initial boot menu will show a new boot entry named "Automated Installation" which gets automatically selected after a 10-second timeout.

Overview

Assistant Tool

The proxmox-auto-install-assistant tool provides the prepare-iso sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation. You will have to install it first: 

apt install proxmox-auto-install-assistant

NOTE: The xorriso binary is required for preparing an ISO. On Debian-based systems you can install it using apt install xorriso.

See the help of the sub-command has more details: proxmox-auto-install-assistant prepare-iso --help.

Answer File Format

The answer file is a TOML-formatted configuration file that provides the basic configuration for a system, like the root password, the network configuration and the target root disk.

In order to allow installation of systems with (for example) many disks and network devices you can use filters to match on device properties, such as serial numbers, vendor or other unique details like the MAC address of a network interface.

For more details about the schema and the possible filters see the Answer File Format section.

Answer File Source

There are several locations where the automatic installer can fetch an answer file from:

  • from the ISO directly
  • read from a separate partition with the PROXMOX-AIS partition label
  • fetched through HTTP from the network
    • the source URL can be pre-determined or queried from DNS or DHCP

For now, you must choose exactly one source.

Prepare an Installation ISO

See the following for a more detailed description of the possible sources and examples on how to prepare a ISO with the different fetch-from modes.

Answer included in the ISO

It is possible to prepare an ISO so that it includes an answer.toml directly, allowing you to have a unified medium for rolling out automatic installation. 

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from iso --answer-file /path/to/answer.toml

Answer on Separate Partition

The ISO can be prepared on a file system labelled proxmox-ais or PROXMOX-AIS (Automated Installation Source) containing the answer.toml file (for example, on a USB flash drive). 

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from partition

Afterwards, prepare the USB flash drive, for example on /dev/sdX1. You may then adapt and run the following commands as root: 

# format as new vfat file system (DESTRUCTIVE!)
mkfs.vfat /dev/sdX1

# set the label
fatlabel /dev/sdX1 "PROXMOX-AIS"

# mount the USB pen drive 
mkdir /mnt/usb
mount /dev/sdX1 /mnt/usb

# copy over the answer file as "answer.toml"
cp my-prepared-answer.toml /mnt/usb/answer.toml

# unmount the USB pen drive
sync
umount /mnt/usb

Answer Fetched via HTTP

In order to fetch the answer file via HTTP(S), the installer needs to know the URL. The URL may be provided via the following ways:

  • URL defined in the ISO
  • as DHCP option (250)
  • via a DNS TXT record
    • The DNS TXT record needs to be located at proxmox-auto-installer.{search domain}, where {search domain} is the search domain provided by the DHCP server.
Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

The HTTP(S) POST request sends JSON data that can help to identify the physical machine and use that information to generate a custom answer file.

You can see how this information looks like by using the sudo proxmox-auto-install-assistant system-info command.

Certificate Fingerprint Matching

It is possible to provide the 'SHA256' certificate fingerprint of the TLS certificate. This is useful in the following situations:

  • the URL is using an IP address instead of a FQDN
  • a self-signed certificate is used that the installer does not trust
  • added security by pinning the known certificate of the server

There are three ways to provide the 'SHA256' fingerprint to the installer:

  • Fingerprint defined in the ISO
  • as a DHCP option (251)
  • via a DNS TXT record
    • The DNS TXT record must be located at proxmox-auto-installer-cert-fingerprint.{search domain}, where {search domain} is the search domain provided by the DHCP server.
Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!

Example

Similarly, it is possible to specify the URL from which the installer should fetch an answer file as well as the TLS certificate fingerprint. For example, to specify both: 

proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from http --url "https://10.0.0.100/get_answer/" --cert-fingerprint "04:42:97:27:F6:29:2F:9F:3D:7F:13:11:C8:E2:F5:5F:84:03:95:D9:F5:14:72:7C:9E:90:47:03:D2:96:2B:EC"

Answer File Format

The answer file is expected in TOML format.

The following example shows a basic answer file that uses the DHCP-provided network settings and will use ZFS in a RAID-1 on disks sda and sdb: 

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
root_ssh_keys = [
    "ssh-ed25519 AAAA..."
]

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
disk_list = ["sda", "sdb"]

Global Section

This section contains the following keys:

  • keyboard -- The keyboard layout with the following possible options:
 de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr
  • country -- The country code in the two letter variant. For example, at, us or fr.
  • fqdn -- The fully qualified domain name of the host. The domain part will be used as the search domain.
  • mailto -- The default email address for the user root.
  • timezone -- The timezone in tzdata format. For example, Europe/Vienna or America/New_York.
  • root_password -- The password for the root user.
  • root_ssh_keys -- Optional. SSH public keys to add to the root users authorized_keys file after the installation.
  • reboot_on_error -- If set to true, the installer will reboot automatically when an error is encountered. The default behavior is to wait to give the administrator a chance to investigate why the installation failed.

Network Section

This section contains the following keys:

  • source -- Where to source the static network configuration from. This can be from-dhcp or from-answer.
    If set to from-dhcp the other network options will be ignored and the installer will use the active NIC and the DHCP settings received during installation to write out a static network configuration.
  • cidr -- The IP address in CIDR notation. For example, 192.168.1.10/24
  • dns -- The IP address of the DNS server.
  • gateway -- The IP address of the default gateway.
  • filter -- Filter against the UDEV properties to select the network card. See filters.

Disk Setup Section

This section contains the following keys:

  • filesystem -- One of the following options: ext4, xfs, zfs, or btrfs.
  • disk_list -- List of disks to use. Useful if you are sure about the disk names. For example: disk_list = ["sda", "sdb"]
  • filter -- Filter against UDEV properties to select the disks for the installation. See filters.
    NOTE: Use either disk_list or filter. Defining both is not allowed.
  • filter_match -- Can be "any" or "all". Decides if a match of any filter is enough of if all filters need to match for a disk to be selected. Default is "any".
  • zfs -- Defines ZFS-specific properties. See ZFS Advanced Options of our documentation. The properties are:
    •  raid -- The RAID level that should be used. Options are raid0, raid1, raid10, raidz-1, raidz-2, or raidz-3.
    •  ashift
    •  arc_max
    •  checksum
    •  compress
    •  copies
    •  hdsize
  • lvm -- Advanced properties that can be used with the ext4 or xfs file system. See LVM Advanced Options. The properties are:
    •  hdsize
    •  swapsize
    •  maxroot
    •  maxvz
    •  minfree
  • btrfs -- Defines BTRFS specific options.
    •  raid -- The RAID level that should be used. Options are raid0, raid1, and raid10.
    •  hdsize

Answer File Validation

The proxmox-auto-install-assistant tool can also be used to validate the syntax of an answer file and display the identifying information that will be sent to the HTTP(s) server when fetching the answer file.

For example, to validate an answer file: 

$ proxmox-auto-install-assistant validate-answer answer.toml 
The file was parsed successfully, no syntax errors found!

Filters

Filters allow you to match against device properties exposed by udevadm.

The proxmox-auto-install-assistant utility can display these properties and allows you to test filters in advance. The utility is available in the installer environment (via its debug mode) and on already existing Proxmox Virtual Environment installation.

For example, to fetch information about the available disks: 

$ proxmox-auto-install-assistant device-info -t disk
[…]
    "nvme1n1": {
      "CURRENT_TAGS": ":systemd:",
      "DEVLINKS": "/dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1 /dev/disk/by-path/pci-0000:e2:00.0-nvme-1 /dev/disk/by-diskseq/12 /dev/disk/by-id/nvme-eui.01000000000000008ce38ee300708529 /dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX",
      "DEVNAME": "/dev/nvme1n1",
      "DEVPATH": "/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1",
      "DEVTYPE": "disk",
      "DISKSEQ": "12",
      "ID_MODEL": "KIOXIA KCMYXVUG1T60",
      "ID_NSID": "1",
      "ID_PART_TABLE_TYPE": "gpt",
      "ID_PART_TABLE_UUID": "539fabeb-aecd-6643-94a9-f28a68cfa12d",
      "ID_PATH": "pci-0000:e2:00.0-nvme-1",
      "ID_PATH_TAG": "pci-0000_e2_00_0-nvme-1",
      "ID_REVISION": "1UETE103",
      "ID_SERIAL": "KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1",
      "ID_SERIAL_SHORT": "9DU0A02D0L33",
      "ID_WWN": "eui.01000000000000008ce38ee300708529",
      "MAJOR": "259",
      "MINOR": "5",
      "SUBSYSTEM": "block",
      "TAGS": ":systemd:",
      "USEC_INITIALIZED": "3169050"
    },
[…]

The key of the filter decides on which property it should be applied to. For example, in order to match against the vendor and model number of the disk, the filter in the answer file could look like this: 

filter.ID_SERIAL = "KIOXIA_KCMYXVUG1T60*"

Note: The * globbing symbol at the end is used to match anything after the defined filter!

This will match for all Kioxia disks with that model number. You may verify which disks will be found by the filter by running the following command: 

$ proxmox-auto-install-assistant device-match disk ID_SERIAL='KIOXIA_KCMYXVUG1T60*'
[
  "nvme0n1",
  "nvme1n1",
  "nvme4n1",
  "nvme5n1"
]

The filter_match parameter controls whether all filters must apply, or if it is enough if any of the filters match. This makes it possible to use different disk models for the installation by using different properties.

For example: 

filter.ID_SERIAL = "KIOXIA*"
filter.ID_MODEL = "ATP*"

Note: For network cards, only the first match will be used as the installer requires only one network card.

More complex network setups can be configured after the installation. Using properties with unique identifiers will result in the most predictable behavior (for example, the MAC address).

Filter Syntax

The following special characters can be used in filters:

  • ? -- matches any single characters
  • * -- matches any number of characters, can be none
  • [a], [abc], [0-9] -- matches any single character inside the brackets, ranges are possible
  • [!a] -- negate the filter, any single character but the ones specified

Useful Properties

For network cards, the following properties can be useful:

  • ID_NET_NAME
  • ID_NET_NAME_MAC
  • ID_VENDOR_FROM_DATABASE
  • ID_MODEL_FROM_DATABASE

For disks, these properties can be useful:

  • DEVNAME
  • ID_SERIAL_SHORT
  • ID_WWNKIOXIA_KCMYXVUG1T60
  • ID_MODEL
  • ID_SERIAL

Examples

Answer Files

ZFS Mirror on Samsung Disks

We assume that there are only two Samsung disks present. These should be used for the OS in a ZFS Mirror (RAID-1). Additionally, we want to use only 150 GiB of the disks' capacities and leave the rest empty for further customization. 

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
zfs.hdsize = 150
filter.ID_MODEL = "Samsung*"

Ext4 With No Swap and Data LV on /dev/sda

Install on /dev/sda and define swap and data LVs with size 0. 

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-dhcp"

[disk-setup]
filesystem = "ext4"
lvm.swapsize = 0
lvm.maxvz = 0
disk_list = ['sda']

ZFS Mirror With Manual Network Config

[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"

[network]
source = "from-answer"
cidr = "10.10.10.10/24"
dns = "10.10.10.1"
gateway = "10.10.10.1"
filter.ID_NET_NAME_MAC = "*e43d1afa379a"

[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
filter.ID_MODEL = "Samsung*"

Note: There is a * in the filter for the network card. This is necessary, because that property usually has enx prefixed.

When displaying this property with proxmox-auto-install-assistant device-info -t network, its full value looks like this: 

"ID_NET_NAME_MAC": "enxe43d1afa379a",

Serving Answer Files via HTTP

Note: These are merely examples! Please ensure that your connection is secured via TLS or that your network is trusted.

Setting up a reverse proxy in front that provides TLS for your server is a good idea.


Serving a Static Answer File via netcat

This variant is doing the bare minimum to provide a single answer file.

Ensure that you have netcat-traditional installed: 

apt update
apt install netcat-traditional

Create a directory in which the server will run and give it adequate permissions, for example: 

mkdir -p /srv/proxmox/auto-install-server
chmod 700 /srv/proxmox/auto-install-server

Place your answer.toml file in that directory.

You may then serve the file on https://[HOST IP]:8000 like this: 

cd /srv/proxmox/auto-install-server
while true; do cat <(printf "HTTP/1.1 200 OK\n\n") answer.toml | nc -l -q 0 -p 8000; done &

The process will run in the background.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

To terminate the process, you can kill it via its job ID or its PID. For such background jobs those can be listed via jobs -l.

For example, if the process's job ID is 1 and its PID is 371012, you can terminate it by running either kill %1 or kill 371012.


Serving a Static Answer File via Python

This option showcases how to use Python to serve a single answer file. It can be a starting point for your solution.

Ensure that you have python3-aiohttp installed: 

apt update
apt install python3-aiohttp

Create a directory in which the server will run and give it adequate permissions, for example: 

mkdir -p /srv/proxmox/auto-install-server
chmod 700 /srv/proxmox/auto-install-server

Place your answer.toml file in that directory.

Then save the following script as server.py in the server's directory as well: 

import logging

from aiohttp import web

routes = web.RouteTableDef()


@routes.post("/answer")
async def answer(request: web.Request):
    logging.info(f"Received request from peer '{request.remote}'")

    file_contents = app.get("answer_file", None)

    if file_contents is None:
        return web.Response(status=404, text="not found")

    return web.Response(text=file_contents)


if __name__ == "__main__":
    app = web.Application()

    with open("answer.toml") as answer_file:
        file_contents = answer_file.read()

    app["answer_file"] = file_contents

    logging.basicConfig(level=logging.INFO)

    app.add_routes(routes)
    web.run_app(app, host="0.0.0.0", port=8000)

The server's directory should now look like this: 

# tree -p /srv/proxmox/auto-install-server
[drwx------]  /srv/proxmox/auto-install-server
├── [-rw-r--r--]  answer.toml
└── [-rw-r--r--]  server.py

1 directory, 2 files

You may now serve the answer file on https://[HOST IP]:8000/answer by starting the server with Python: 

cd /srv/proxmox/auto-install-server
python3 server.py

The server will run in the foreground and can be terminated by hitting CTRL+C in the console.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

Serving Answer Files Depending on MAC Address via Python

This is a slightly more advanced example that can dynamically serve answer files depending on the MAC address of the host that made the request. This is achieved by reading the JSON data from the HTTP POST request.

Ensure that you have python3-aiohttp and python3-tomlkit installed: 

apt update
apt install python3-aiohttp python3-tomlkit

Create a directory in which the server will run and give it adequate permissions, for example: 

mkdir -p /srv/proxmox/auto-install-server
chmod 700 /srv/proxmox/auto-install-server

Before you can run the server below, you must set up the following:

  • Place a file named default.toml in the server's directory.
This is the fallback answer file that will be used if no MAC address match was found.
  • Create a directory named answers in the server's directory.
This directory will contain the answer files associated with each MAC address.

You may then add as many answer files to the answers directory as you want.

  • Each file must be named after the MAC address it is associated with and also end with .toml.
For example: BC:24:11:AB:12:21.toml

Then save the following script as server.py in the server's directory as well: 

import logging
import json
import pathlib

try:
    import tomlkit
    from aiohttp import web
except ImportError as e:
    import sys

    message = """Could not import required packages.
Please ensure you've installed all necessary packages first!

On Debian-based distributions, you should be able to install them via:

\tapt update
\tapt install python3-aiohttp python3-tomlkit"""

    print(message, file=sys.stderr)

    raise e

DEFAULT_ANSWER_FILE_PATH = pathlib.Path("./default.toml")
ANSWER_FILE_DIR = pathlib.Path("./answers/")

routes = web.RouteTableDef()


@routes.post("/answer")
async def answer(request: web.Request):
    try:
        request_data = json.loads(await request.text())
    except json.JSONDecodeError as e:
        return web.Response(
            status=500,
            text=f"Internal Server Error: failed to parse request contents: {e}",
        )

    logging.info(
        f"Request data for peer '{request.remote}':\n"
        f"{json.dumps(request_data, indent=1)}"
    )

    try:
        answer = create_answer(request_data)

        logging.info(f"Answer file for peer '{request.remote}':\n{answer}")

        return web.Response(text=answer)
    except Exception as e:
        logging.exception(f"failed to create answer: {e}")
        return web.Response(status=500, text=f"Internal Server Error: {e}")


def create_answer(request_data: dict) -> str:
    with open(DEFAULT_ANSWER_FILE_PATH) as file:
        answer = tomlkit.parse(file.read())

    for nic in request_data.get("network_interfaces", []):
        if "mac" not in nic:
            continue

        answer_mac = lookup_answer_for_mac(nic["mac"])
        if answer_mac is not None:
            answer = answer_mac

    return tomlkit.dumps(answer)


def lookup_answer_for_mac(mac: str) -> tomlkit.TOMLDocument | None:
    mac = mac.lower()

    for filename in ANSWER_FILE_DIR.glob("*.toml"):
        if filename.name.lower().startswith(mac):
            with open(filename) as mac_file:
                return tomlkit.parse(mac_file.read())


def assert_default_answer_file_exists():
    if not DEFAULT_ANSWER_FILE_PATH.exists():
        raise RuntimeError(
            f"Default answer file '{DEFAULT_ANSWER_FILE_PATH}' does not exist"
        )


def assert_default_answer_file_parseable():
    with open(DEFAULT_ANSWER_FILE_PATH) as file:
        try:
            tomlkit.parse(file.read())
        except Exception as e:
            raise RuntimeError(
                "Could not parse default answer file "
                f"'{DEFAULT_ANSWER_FILE_PATH}':\n{e}"
            )


def assert_answer_dir_exists():
    if not ANSWER_FILE_DIR.exists():
        raise RuntimeError(f"Answer file directory '{ANSWER_FILE_DIR}' does not exist")


if __name__ == "__main__":
    assert_default_answer_file_exists()
    assert_answer_dir_exists()
    assert_default_answer_file_parseable()

    app = web.Application()

    logging.basicConfig(level=logging.INFO)

    app.add_routes(routes)
    web.run_app(app, host="0.0.0.0", port=8000)

The server's directory should now look like this: 

# tree -p /srv/proxmox/auto-install-server
[drwx------]  /srv/proxmox/auto-install-server
├── [drwxr-xr-x]  answers
│   ├── [-rw-r--r--]  BC:24:11:AB:12:21.toml
│   ├── [-rw-r--r--]  BC:24:11:BE:F2:A2.toml
│   └── [-rw-r--r--]  BC:24:11:DC:CD:21.toml
├── [-rw-r--r--]  default.toml
└── [-rw-r--r--]  server.py

2 directories, 5 files

You may now serve your answer files on https://[HOST IP]:8000/answer by starting the server with Python: 

cd /srv/proxmox/auto-install-server
python3 server.py

The server will run in the foreground and can be terminated by hitting CTRL+C in the console.

Note: Please ensure that your connection is secured by TLS through something like a reverse proxy.

Third party tools

The following third-party tools around the Proxmox Automated Installation might be useful:

Troubleshooting

If the installation fails for some reason, it will drop into a shell (unless the reboot_on_error option in the answer file is set to true). This gives you the chance to troubleshoot what went wrong.

The log files of interest will most likely be:

  • /tmp/fetch_answer.log - the steps to retrieve an answer file
  • /tmp/auto_installer - parsing of the answer file, matching of hardware to use
  • /tmp/install-low-level-start-session.log - the actual installation process
Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Automated_Installation&oldid=12000"