Automated Installation: Difference between revisions
No edit summary |
A.lauterer (talk | contribs) (→Assistant Tool: add command to install it) |
||
(One intermediate revision by one other user not shown) | |||
Line 17: | Line 17: | ||
=== Assistant Tool === | === Assistant Tool === | ||
The <code>proxmox-auto-install-assistant</code> tool provides the <code>prepare-iso</code> sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation.<!-- TODO: You can download it from... upload static compiled version and place link here. --> | The <code>proxmox-auto-install-assistant</code> tool provides the <code>prepare-iso</code> sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation. You will have to install it first: | ||
<pre>apt install proxmox-auto-install-assistant</pre><!-- TODO: You can download it from... upload static compiled version and place link here. --> | |||
'''NOTE:''' The <code>xorriso</code> binary is required for preparing an ISO. On Debian-based systems you can install it using <code>apt install xorriso</code>. | '''NOTE:''' The <code>xorriso</code> binary is required for preparing an ISO. On Debian-based systems you can install it using <code>apt install xorriso</code>. | ||
Line 378: | Line 379: | ||
'''Note:''' There is a <code>*</code> in the filter for the network card. This is necessary, because that property usually has <code>enx</code> prefixed. | '''Note:''' There is a <code>*</code> in the filter for the network card. This is necessary, because that property usually has <code>enx</code> prefixed. | ||
When displaying this property with <code>proxmox- | When displaying this property with <code>proxmox-auto-install-assistant device-info -t network</code>, its full value looks like this: | ||
<syntaxhighlight lang="json"> | <syntaxhighlight lang="json"> |
Revision as of 23:11, 26 April 2024
This article documents the automated installation process of Proxmox projects like Proxmox VE.
Introduction
The automated installation method allows installing a Proxmox solution in an unattended manner. This enables you to fully automate the setup process on bare-metal. Once the installation is complete and the host has booted up, automation tools like Ansible can be used to further configure the installation.
The necessary options for the installer must be provided in an answer file. This file allows using filter rules to determine which disks and network cards should be used.
In order to use the automated installation it is first necessary to choose a source from which the answer file is fetched from and then prepare an installation ISO with that choice.
Once the ISO is prepared, its initial boot menu will show a new boot entry named "Automated Installation" which gets automatically selected after a 10-second timeout.
Overview
Assistant Tool
The proxmox-auto-install-assistant
tool provides the prepare-iso
sub-command which can be used to prepare a new enough, but otherwise standard ISO of a Proxmox project for automated installation. You will have to install it first:
apt install proxmox-auto-install-assistant
NOTE: The xorriso
binary is required for preparing an ISO. On Debian-based systems you can install it using apt install xorriso
.
See the help of the sub-command has more details: proxmox-auto-install-assistant prepare-iso --help
.
Answer File Format
The answer file is a TOML-formatted configuration file that provides the basic configuration for a system, like the root password, the network configuration and the target root disk.
In order to allow installation of systems with (for example) many disks and network devices you can use filters to match on device properties, such as serial numbers, vendor or other unique details like the MAC address of a network interface.
For more details about the schema and the possible filters see the Answer File Format section.
Answer File Source
There are several locations where the automatic installer can fetch an answer file from:
- from the ISO directly
- read from a separate partition with the
PROXMOX-AIS
partition label - fetched through HTTP from the network
- the source URL can be pre-determined or queried from DNS or DHCP
For now, you must choose exactly one source.
Prepare an Installation ISO
See the following for a more detailed description of the possible sources and examples on how to prepare a ISO with the different fetch-from modes.
Answer included in the ISO
It is possible to prepare an ISO so that it includes an answer.toml
directly, allowing you to have a unified medium for rolling out automatic installation.
proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from iso --answer-file /path/to/answer.toml
Answer on Separate Partition
The ISO can be prepared on a file system labelled proxmox-ais
or PROXMOX-AIS
(Automated Installation Source) containing the answer.toml
file (for example, on a USB flash drive).
proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from partition
Afterwards, prepare the USB flash drive, for example on /dev/sdX1
. You may then adapt and run the following commands as root
:
# format as new vfat file system (DESTRUCTIVE!)
mkfs.vfat /dev/sdX1
# set the label
fatlabel /dev/sdX1 "PROXMOX-AIS"
# mount the USB pen drive
mkdir /mnt/usb
mount /dev/sdX1 /mnt/usb
# copy over the answer file as "answer.toml"
cp my-prepared-answer.toml /mnt/usb/answer.toml
# unmount the USB pen drive
sync
umount /mnt/usb
Answer Fetched via HTTP
In order to fetch the answer file via HTTP(S), the installer needs to know the URL. The URL may be provided via the following ways:
- URL defined in the ISO
- as DHCP option (
250
) - via a DNS TXT record
- The DNS TXT record needs to be located at
proxmox-auto-installer.{search domain}
, where{search domain}
is the search domain provided by the DHCP server.
- The DNS TXT record needs to be located at
- Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!
The HTTP(S) POST request sends JSON data that can help to identify the physical machine and use that information to generate a custom answer file.
You can see how this information looks like by using the sudo proxmox-auto-install-assistant system-info
command.
Certificate Fingerprint Matching
It is possible to provide the 'SHA256' certificate fingerprint of the TLS certificate. This is useful in the following situations:
- the URL is using an IP address instead of a FQDN
- a self-signed certificate is used that the installer does not trust
- added security by pinning the known certificate of the server
There are three ways to provide the 'SHA256' fingerprint to the installer:
- Fingerprint defined in the ISO
- as a DHCP option (
251
) - via a DNS TXT record
- The DNS TXT record must be located at
proxmox-auto-installer-cert-fingerprint.{search domain}
, where{search domain}
is the search domain provided by the DHCP server.
- The DNS TXT record must be located at
- Note: Fetching the fingerprint via DHCP or DNS records is only done if the same method is used to retrieve the URL!
Example
Similarly, it is possible to specify the URL from which the installer should fetch an answer file as well as the TLS certificate fingerprint. For example, to specify both:
proxmox-auto-install-assistant prepare-iso /path/to/source.iso --fetch-from http --url "https://10.0.0.100/get_answer/" --cert-fingerprint "04:42:97:27:F6:29:2F:9F:3D:7F:13:11:C8:E2:F5:5F:84:03:95:D9:F5:14:72:7C:9E:90:47:03:D2:96:2B:EC"
Answer File Format
The answer file is expected in TOML format.
The following example shows a basic answer file that uses the DHCP-provided network settings and will use ZFS in a RAID-1 on disks sda
and sdb
:
[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
root_ssh_keys = [
"ssh-ed25519 AAAA..."
]
[network]
source = "from-dhcp"
[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
disk_list = ["sda", "sdb"]
Global Section
This section contains the following keys:
keyboard
-- The keyboard layout with the following possible options:
de de-ch dk en-gb en-us es fi fr fr-be fr-ca fr-ch hu is it jp lt mk nl no pl pt pt-br se si tr
country
-- The country code in the two letter variant. For example,at
,us
orfr
.fqdn
-- The fully qualified domain name of the host. The domain part will be used as the search domain.mailto
-- The default email address for the userroot
.timezone
-- The timezone intzdata
format. For example,Europe/Vienna
orAmerica/New_York
.root_password
-- The password for theroot
user.root_ssh_keys
-- Optional. SSH public keys to add to theroot
usersauthorized_keys
file after the installation.reboot_on_error
-- If set totrue
, the installer will reboot automatically when an error is encountered. The default behavior is to wait to give the administrator a chance to investigate why the installation failed.
Network Section
This section contains the following keys:
source
-- Where to source the static network configuration from. This can befrom-dhcp
orfrom-answer
.- If set to
from-dhcp
the other network options will be ignored and the installer will use the active NIC and the DHCP settings received during installation to write out a static network configuration.
- If set to
cidr
-- The IP address in CIDR notation. For example,192.168.1.10/24
dns
-- The IP address of the DNS server.gateway
-- The IP address of the default gateway.filter
-- Filter against theUDEV
properties to select the network card. See filters.
Disk Setup Section
This section contains the following keys:
filesystem
-- One of the following options:ext4
,xfs
,zfs
, orbtrfs
.disk_list
-- List of disks to use. Useful if you are sure about the disk names. For example:disk_list = ["sda", "sdb"]
filter
-- Filter againstUDEV
properties to select the disks for the installation. See filters.- NOTE: Use either
disk_list
orfilter
. Defining both is not allowed.
- NOTE: Use either
filter_match
-- Can be "any" or "all". Decides if a match of any filter is enough of if all filters need to match for a disk to be selected. Default is "any".zfs
-- Defines ZFS-specific properties. See ZFS Advanced Options of our documentation. The properties are:-
raid
-- The RAID level that should be used. Options areraid0
,raid1
,raid10
,raidz-1
,raidz-2
, orraidz-3
. -
ashift
-
arc_max
-
checksum
-
compress
-
copies
-
hdsize
-
lvm
-- Advanced properties that can be used with theext4
orxfs
file system. See LVM Advanced Options. The properties are:-
hdsize
-
swapsize
-
maxroot
-
maxvz
-
minfree
-
btrfs
-- Defines BTRFS specific options.-
raid
-- The RAID level that should be used. Options areraid0
,raid1
, andraid10
. -
hdsize
-
Answer File Validation
The proxmox-auto-install-assistant
tool can also be used to validate the syntax of an answer file and display the identifying information that will be sent to the HTTP(s) server when fetching the answer file.
For example, to validate an answer file:
$ proxmox-auto-install-assistant validate-answer answer.toml
The file was parsed successfully, no syntax errors found!
Filters
Filters allow you to match against device properties exposed by udevadm
.
The proxmox-auto-install-assistant
utility can display these properties and allows you to test filters in advance. The utility is available in the installer environment (via its debug mode) and on
already existing Proxmox Virtual Environment installation.
For example, to fetch information about the available disks:
$ proxmox-auto-install-assistant device-info -t disk
[…]
"nvme1n1": {
"CURRENT_TAGS": ":systemd:",
"DEVLINKS": "/dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1 /dev/disk/by-path/pci-0000:e2:00.0-nvme-1 /dev/disk/by-diskseq/12 /dev/disk/by-id/nvme-eui.01000000000000008ce38ee300708529 /dev/disk/by-id/nvme-KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX",
"DEVNAME": "/dev/nvme1n1",
"DEVPATH": "/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1",
"DEVTYPE": "disk",
"DISKSEQ": "12",
"ID_MODEL": "KIOXIA KCMYXVUG1T60",
"ID_NSID": "1",
"ID_PART_TABLE_TYPE": "gpt",
"ID_PART_TABLE_UUID": "539fabeb-aecd-6643-94a9-f28a68cfa12d",
"ID_PATH": "pci-0000:e2:00.0-nvme-1",
"ID_PATH_TAG": "pci-0000_e2_00_0-nvme-1",
"ID_REVISION": "1UETE103",
"ID_SERIAL": "KIOXIA_KCMYXVUG1T60_9DUXXXXXXXX_1",
"ID_SERIAL_SHORT": "9DU0A02D0L33",
"ID_WWN": "eui.01000000000000008ce38ee300708529",
"MAJOR": "259",
"MINOR": "5",
"SUBSYSTEM": "block",
"TAGS": ":systemd:",
"USEC_INITIALIZED": "3169050"
},
[…]
The key of the filter decides on which property it should be applied to. For example, in order to match against the vendor and model number of the disk, the filter in the answer file could look like this:
filter.ID_SERIAL = "KIOXIA_KCMYXVUG1T60*"
Note: The *
globbing symbol at the end is used to match anything after the defined filter!
This will match for all Kioxia disks with that model number. You may verify which disks will be found by the filter by running the following command:
$ proxmox-auto-install-assistant device-match disk ID_SERIAL='KIOXIA_KCMYXVUG1T60*'
[
"nvme0n1",
"nvme1n1",
"nvme4n1",
"nvme5n1"
]
The filter_match
parameter controls whether all filters must apply, or if it is enough if any of the filters match. This makes it possible to use different disk models for the installation by using different properties.
For example:
filter.ID_SERIAL = "KIOXIA*"
filter.ID_MODEL = "ATP*"
Note: For network cards, only the first match will be used as the installer requires only one network card.
More complex network setups can be configured after the installation. Using properties with unique identifiers will result in the most predictable behavior (for example, the MAC address).
Filter Syntax
The following special characters can be used in filters:
?
-- matches any single characters*
-- matches any number of characters, can be none[a]
,[abc]
,[0-9]
-- matches any single character inside the brackets, ranges are possible[!a]
-- negate the filter, any single character but the ones specified
Useful Properties
For network cards, the following properties can be useful:
ID_NET_NAME
ID_NET_NAME_MAC
ID_VENDOR_FROM_DATABASE
ID_MODEL_FROM_DATABASE
For disks, these properties can be useful:
DEVNAME
ID_SERIAL_SHORT
ID_WWNKIOXIA_KCMYXVUG1T60
ID_MODEL
ID_SERIAL
Examples
Answer Files
ZFS Mirror on Samsung Disks
We assume that there are only two Samsung disks present. These should be used for the OS in a ZFS Mirror (RAID-1). Additionally, we want to use only 150 GiB of the disks' capacities and leave the rest empty for further customization.
[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
[network]
source = "from-dhcp"
[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
zfs.hdsize = 150
filter.ID_MODEL = "Samsung*"
Ext4 With No Swap and Data LV on /dev/sda
Install on /dev/sda
and define swap and data LVs with size 0
.
[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
[network]
source = "from-dhcp"
[disk-setup]
filesystem = "ext4"
lvm.swapsize = 0
lvm.maxvz = 0
disk_list = ['sda']
ZFS Mirror With Manual Network Config
[global]
keyboard = "de"
country = "at"
fqdn = "pveauto.testinstall"
mailto = "mail@no.invalid"
timezone = "Europe/Vienna"
root_password = "123456"
[network]
source = "from-answer"
cidr = "10.10.10.10/24"
dns = "10.10.10.1"
gateway = "10.10.10.1"
filter.ID_NET_NAME_MAC = "*e43d1afa379a"
[disk-setup]
filesystem = "zfs"
zfs.raid = "raid1"
filter.ID_MODEL = "Samsung*"
Note: There is a *
in the filter for the network card. This is necessary, because that property usually has enx
prefixed.
When displaying this property with proxmox-auto-install-assistant device-info -t network
, its full value looks like this:
"ID_NET_NAME_MAC": "enxe43d1afa379a",
Troubleshooting
If the installation fails for some reason, it will drop into a shell (unless the reboot_on_error
option in the answer file is set to true
). This gives you the chance to troubleshoot what went wrong.
The log files of interest will most likely be:
/tmp/fetch_answer.log
- the steps to retrieve an answer file/tmp/auto_installer
- parsing of the answer file, matching of hardware to use/tmp/install-low-level-start-session.log
- the actual installation process