Roadmap

~~Offline updates~~ done
~~Cross-cluster migration mechanism~~ - foundation and CLI integration released with Proxmox VE 7.3
Enhancement and stabilization of the integrated Software Defined Network solution
Project "Cattle and Pets"
- Improve user and virtual machine managing experience on big setups
- Add cluster-wide update and status control center
Cluster Resource Scheduling Improvements
Short/Mid-Term:
- ~~Re-balance service on fresh start up (request-stop to request-start configuration change)~~ released with Proxmox VE 7.4
- Account for non-HA virtual guests
Mid/Long-Term:
- Add Dynamic-Load scheduling mode
- Add option to schedule non-HA virtual guests too

Release History

Proxmox VE 8.0 beta1

Released 9. June 2023: See Downloads

Note: this is a test version not meant for production use yet.

Based on Debian 12 Bookworm (testing)
Latest 6.2 Kernel as stable default
QEMU 8.0.2
LXC 5.0.2
ZFS 2.1.11
Ceph Quincy 17.2.6

Highlights

New major release based on the great Debian Bookworm.
Seamless upgrade from Proxmox VE 7.4, see Upgrade from 7 to 8
Ceph Quincy enterprise repository.
Access the most stable Ceph repository through any Proxmox VE subscription.
Add access realm sync jobs.
Synchronize users and groups from an LDAP/AD server automatically at regular intervals.
Integrate host network bridge and VNet access when configuring virtual guests into Proxmox VE's ACL system.
With the new SDN.Use privilege and the new /sdn/zones/<zone>/<bridge-or-vnet>/<vlan-tag> ACL object path, one can give out fine-grained usage permissions for specific networks to users.

Changelog Overview

Enhancements in the web interface (GUI)

The Ceph repository selection now takes into account the subscription status of the complete cluster and recommends the optimal version for the cluster.
Improved Dark color theme:

The Dark color theme, introduced in Proxmox VE 7.4, received a lot of feedback from our community, which resulted in further improvements.

Set strict SameSite attribute on the Authorization cookie
The Markdown parser, used in notes, has been improved:
- it allows setting the target for links, to make any link open in a new tab or window.
- it allows providing URLs with a scheme different from HTTP/HTTPS;
  You can now directly link to resources like rdp://<rest-of-url>, providing convenience links in the guest notes.
- tag-names and protocols are matched case-insensitive.
The mobile UI code was refactored to not suffer from incompatible changes made for the web-based GUI.
The generated CSR used by the built-in ACME client now sets the correct CSR version (0 instead of 2).
Uploading files now only computes the MD5 sum of the uploaded file, if it can be used for comparison with the user-provided one.
Firewall settings: Improve the alignment of permissions checked by the web UI with the permissions actually required by the API.
Explicitly disallow internal-only tmpfilename parameter for file uploads.
Fix multipart HTTP uploads without Content-Type header.
Show Ceph pool number in the web UI, as it is often mentioned in Ceph warnings and errors.
Improved translations, among others:
- Ukrainian (NEW)
- Japanese
- Simplified Chinese
- Traditional Chinese
- The size units (Bytes, KB, MiB,...) are now passed through the translation framework as well, allowing localized variants (e.g., for French).
- The language selection is now localized and displayed in the currently selected language

Virtual Machines (KVM/QEMU)

New QEMU Version 8.0:
- The virtiofsd codebase was replaced by a new and improved implementation based on Rust, which is packaged separately.
- QEMU Guest Agent now has initial support for NetBSD and OpenBSD.
- Many more changes, see the upstream changelog for details.

Avoid invalid smm machine flag for aarch64 VM when using serial display and SeaBIOS.
Warn if a network interface is not connected to a bridge on VM startup. This can happen if the user manually edited the VM config.
Fix an issue with the improved code for volume size information gathering for passed through disks during backup preparation.
Workaround breaking driver changes in newer Nvidia grid drivers, which prevented mediated devices (mdev) to be reclaimed upon guest exit.
Prefer an explicit configured SMBIOS UUID for Nvidia vGPU passthrough.
If a uuid command line parameter is present, it will be preferred over the default auto-generated UUID, containing the VMID and mdev index.

This fixes an issue, with software inside the guest, which relies on a specific and unique UUID setting.

Improved gathering of current setting for live memory unplugging.
Avoid sending a temporary size of zero to QEMU when resizing block devices. Previously, this was done when resizing RBD volumes, but it is not necessary anymore.
When resizing a disk, spawn a worker task to avoid HTTP request timeout (issue 2315).
Allow resizing qcow2 disk images with snapshots (issue 517).
cloud-init improvements:
- Introduce ciupgrade option that controls whether machines should upgrade packages on boot (issue 3428).
- Better align privilege checks in the web UI with the actual privileges required in the backend.
- Fix an issue where the hostname was not properly set on Fedora/CentOS machines, by passing the hostname via the fqdn option.
- Fix an issue where displaying pending changes via qm and pvesh caused an error.
- Allow setting network options with VM.Config.Cloudinit privileges, instead of requiring the more powerful VM.Config.Network privilege.
Drop unused QMP commands for getting the link and creating/deleting internal snapshots.
Replace usages of deprecated -no-hpet QEMU option with the hpet=off machine flag.

Containers (LXC)

Improve handling of /etc/machine-id on clone operations - the file is now only truncated, if the source did not explicitly set it to 'uninitialized' or remove it. Thus, the admin can decide if they want first-boot semantics or not (see machine-id (5)).
Set memory.high cgroup limit to 99.6% of configured memory. This setting gives the container a chance to free memory before directly running into an Out-of-Memory (OOM) condition. It is applied on lxc.conf generation and on hot-plugging memory to a running container.
Warn users on conflicting, manual, lxc.idmap entries.
Custom mappings can become quite complicated and cause overlaps fast.

By issuing a warning upon container start, the user should find the wrong entry directly.
When resizing a disk, perform plausibility checks already before spawning the worker task. This allows invalid requests to fail earlier.
General code improvements, adhering to best practices for Perl code.

General improvements for virtual guests

When cloning guests, the validation of the provided name of the clone is now happening in the frontend, improving UX.
Add config files in /etc/pve/mapping and privileges Mapping.* in preparation for cluster-wide mapping of PCI/USB devices.

HA Manager

Stability improvements of manual maintenance mode:
- Fix an issue where a request for enabling maintenance mode on a node is lost, in case the rebooted node is the current active Cluster Resource Manager (CRM).
- Fix an issue where a shutdown policy other than migrate could cause a node in maintenance mode to leave maintenance mode too early or fence itself.
- Fix an issue where ha-rebalance-on-start could cause a newly added and already-running service to be shut down and migrated to another node.
  Now, ha-rebalance-on-start ignores services that are already running.
When enabling or disabling maintenance mode via the CLI, the ha-manager command now checks whether the provided node exists.
This avoids misconfigurations, e.g., due to a typo in the node name.

Improved management for Proxmox VE clusters

The rsync invocation used when joining nodes via ssh, which is deprecated, has been adapted to changes in rsync CLI argument parsing in Bookworm.

Backup/Restore

Improve performance of backups that use zstd on fast disks, by invoking zstd without the --rsyncable flag (issue 4605).
Suppress harmless but confusing "storing login ticket failed" errors when backing up to Proxmox Backup Server.
When restoring from backups via the web UI, the VM/CT name is now validated client-side before sending an API request. This helps catching invalid names early.
The web UI now sorts backups by date, whereas it previously sorted backups first by VMID and then by date. The VMID is added as an extra column for users who would like to restore the previous sorting order (issue 4678).
Fix an issue where the backup job editor window occasionally did not show the selected guests (issue 4627).
The fs-freeze-on-backup option of the QEMU guest agent, which controls whether the filesystem should be frozen for backups, can now be set in the web UI.
Improve permission model for backup jobs: Editing backup jobs now generally requires the Datastore.Allocate privilege on the target storage, and editing backup jobs with dumpdir requires root privileges.
Clarify description of the ionice setting.

Storage

The file-based storage-types have two new config options create-base-path and create-subdirs. They replace the mkdir option and separate two different concepts:

create-base-path decides if the path to the storage should be created if it does not exist,

create-subdirs decides if the content-specific sub-directories (guest images, ISO, container template, backups) should be created.

Conflating both settings in the single mkdir option caused a few unwanted effects in certain situations (issue 3214).

The CIFS storage type can now be configured with custom mount options, as it was already possible for the NFS storage type.
The subdir option of the CIFS storage type can now be configured in the web interface. The option can be used to mount a subdirectory of a SMB/CIFS share and was previously only accessible via the API/CLI.
Improve API documentation for the upload method.
The API now allows to also query replication jobs that are disabled.
Allow @ in directory storage path, as it is often used to signify Btrfs subvolumes.
When resizing RBD volumes, always round up sizes to the nearest integer. This avoids errors due to passing a floating-point size to the RBD tooling.

Ceph

Add support for new Ceph enterprise repositories. When installing Ceph via pveceph install or the web UI, you can now choose between the test, no-subscription and enterprise (default) repositories. The -test-repository option of the pveceph install command was removed.
Add pveceph osddetails command to show information about OSDs on the command line, with a level of detail that is comparable to the web UI/API.
Drop support for Ceph Octopus and Pacific, as they are not supported in Proxmox VE 8.
Remove overly restrictive validation of public_network during monitor creation. Configuring a public network like 0::/0 or 0::/1 caused a superfluous "value does not look like a valid CIDR network" error.
The Ceph installation wizard in the web UI does not create monitors and managers called localhost anymore and uses the actual node name instead.

Access Control

Add possibility to define realm sync jobs in the web UI. These allow to synchronize users and groups from an LDAP server automatically at regular intervals.
Add TFA/TOTP lockout to protect against an attacker who has obtained the user password and attempts to guess the second factor:
- If TFA failed too many times in a row, lock this user account out of TFA for an hour. If TOTP failed too many times in a row, disable TOTP for the user account. Using a recovery key will unlock a user account.
- Add pveum tfa unlock command and /access/users/{userid}/unlock-tfa API endpoint for manually unlocking users.
- Add TFA lockout status to responses of /access/tfa and /access/users endpoints.
Fix validity check for LDAP base DNs that was overly strict starting from Proxmox VE 7.4. For example, the check rejected base DNs containing both dashes and spaces (issue 4609).
When authenticating via PAM, pass the PAM_RHOST item. With this, it is possible to manually configure PAM such that certain users (for example root@pam) can only log in from certain hosts.
Add pveum tfa list command for listing second factors on the command line.
The access/ticket API endpoint does not support the deprecated login API (using new-format=0) anymore.
Remove the Permission.Modify privilege from the PVESysAdmin and PVEAdmin roles and restrict it to the Administrator role. This reduces the chances of accidentally granting privilege modification privileges.
Login with TFA: In order to improve UX, fix wording of messages related to recovery keys.
Forbid creating roles with names starting with PVE to reserve these role names for use in future upgrades.
SDN.Use is required on a bridge/vnet (or its zone) in order to configure it in a guest vNIC.
- use /sdn/zones/localnetwork or /sdn/zones/localnetwork/<bridge> to allow usage of all or specific local bridges.
- use /sdn/zones/<zone> or /sdn/zones/<zone>/<bridge> to allow usage of all or specific vnets in a given SDN zone.
Users with VM.Allocate/Datastore.Allocate/Pool.Allocate privileges, but without the Permissions.Modify privilege, can now only assign a subset of their own privileges to specific VM/storage/pool paths, instead of arbitrary roles.

Firewall & Software Defined Networking

Allow to distinguish IP sets and aliases with the same name defined on the datacenter level and on the guest level by providing an explicit prefix (issue 4556). Previously, the innermost IP set/alias took precedence, which is still the default behavior if no prefix is provided.
Fix an issue where an allowed special ICMP-type could accidentally be added as destination port for a layer 4 protocol, breaking firewall rule loading.
Fix setting the correct vlan-protocol for QinQ zones if the bridge is vlan-aware.(issue 4683
Fix an issue where routing between zones was enabled by default in exit nodes. This has been fixed by adding null-routes for each other zone prefix to each zone (issue 4389).
Correctly order vrf and router bgp vrf entries by vrf name in the frr configuration. (issue 4662)
For setups where a node is primary exit node for one vrf and secondary exit for a different vrf, the configuration now also adds the second vrf's default route. (issue 4657)
Allow specifying a custom vxlan-tunnel port per interface.
Update the frr configuration generation to the version of frr shipped in Debian Bookworm.
Fix an issue where reloading the network configuration on a remote node created an error, which hid the actual issue with the network configuration.
Add support for IPv6 SLAAC and router advertisement configuration in /etc/network/interfaces to ifupdown2.
Fix live reloading when changing VLAN and VXLAN specific attributes.
Add support for creating an OVS bridge which tags traffic with a specific VLAN tag to ifupdown2.

This is to match the possibility in ifupdown.

Improvements for the management of Proxmox VE Nodes

pve7to8 compatibility check script added.

As with previous major upgrades, Proxmox VE 7 ships a script checking for issues with the current node/cluster. It should point to any issues which might prevent a successful major upgrade.

Outdated pve6to7 compatibility check script was removed.
Fix an issue where the web UI would display no APT repositories during a major upgrade.
The new version of grub2 provided by Debian Bookworm (2.06-13) fixes an issue where a host using LVM would fail to boot with a message disk `lvmid/...` not found, even though the LVM setup is healthy.

Installation ISO

The version of BusyBox shipped with the ISO was updated to version 1.36.1.
The Proxmox-provided Ceph Quincy repo will be set-up by default, providing updates for a modern Ceph client even if Proxmox Ceph hyper-converged setup is not in use.
Detection of unreasonable system time.

If the system time is older than the time the installer was created, the system notifies the user with a warning.

ethtool is now shipped with the ISO and installed on all systems.
systemd-boot is provided by its own package instead of systemd in Debian Bookworm and is installed with the new ISO.

Notable bug fixes and general improvments

Most git repositories now have a dsc Makefile target to create a Debian Source Package and additionally a sbuild target to create the source package and build it using sbuild.

Known Issues & Breaking Changes

Storage activation now checks that every content type uses a different directory, in order to prevent unexpected interactions between different content types. This breaks setups in which the content-dirs option was set up to map different content types to the same directory, and setups in which some content directories were manually set up as symlinks to a common directory.
QEMU 8.0 removed some previously deprecated features. Proxmox VE 8 won't use the -chardev tty and -chardev parport aliases anymore, and no other features were used by the Proxmox VE stack. Thus, only installations using args inside their guest configs need to check the compatibility. See the Qemu changelog on the topic for details.
- The removed features in QEMU 8.0 also include the Proxmox VE-specific, but unused/deprecated QMP commands get_link_status, snapshot-drive and delete-drive-snapshot.
The lxc.id_map configuration key has been deprecated for a long time by lxc and was replaced by lxc.idmap. With this release, its presence is considered an error. The key can only be present if it was manually added to a guest configuration.
The lxcfs is now built with fuse 3. This upgrade is done on a major release, since all running containers need to be restarted afterwards.
Permission System
- There is a new SDN.Use privilege (and corresponding PVESDNUser role) that is required to configure virtual NICs in guests. See SDN section above for details!
- The Permission.Modify privilege has been removed from the PVESysAdmin and PVEAdmin roles, in order to reduce the chances of accidentally granting the privilege to modify privileges. If a particular setup requires a role with this privilege, it is necessary to define a new custom role and use that instead of PVESysAdmin/PVEAdmin.
- Users with VM.Allocate/Datastore.Allocate/Pool.Allocate privileges, but without the Permissions.Modify privilege, can now only assign a subset of their own privileges to specific VM/storage/pool paths. Previously they could assign any role to specific VM/storage/pool paths. As the privileges usable on specific VM/storage/pool paths were quite limited, this did not allow privilege escalation, but restricting the capabilities now allows adding more powerful privileges in future versions without breaking changes.
Editing backup jobs now generally requires the Datastore.Allocate privilege on the target storage, and editing backup jobs with dumpdir requires root privileges.
User accounts will now be locked after too many attempts to authenticate with a second factor. This is intended to protect against an attacker who has obtained the user password and attempts to guess the second factor. Unlocking requires either a successful login with a recovery key or a manual unlock by an administrator.
The API can handle array-type data differently, while staying backward compatible.
Instead of being able to pass the individual elements separated by null bytes, you can pass the data directly as array.

Proxmox VE 7.4

Released 23. March 2023: See Downloads

Based on Debian Bullseye (11.6)
Latest 5.15 Kernel as stable default
Newer 6.2 kernel as opt-in
QEMU 7.2
LXC 5.0.2
ZFS 2.1.9
Ceph Quincy 17.2.5
Ceph Pacific 16.2.11

Highlights

Proxmox VE now provides a dark theme for the web interface.
Guests in resource tree can now be sorted by their name, not only VMID.
The HA Cluster Resource Scheduler (CRS) stack was expanded to rebalance VMs & CTs automatically on start, not only recovery.
Added CRM command to the HA manager to switch an online node manually into maintenance mode (without reboot.

Changelog Overview

Enhancements in the web interface (GUI)

Add a fully-integrated "Proxmox Dark" color theme variant of the long-time Crisp light theme.

By default, the prefers-color-scheme media query from the Browser/OS will be used to decide the default color scheme.

Users can override the theme via a newly added Color Theme menu in the user menu.

Add "Proxmox Dark" color theme to the Proxmox VE reference documentation.

The prefers-color-scheme media query from the Browser/OS will be used to decide if the light or dark color scheme should be used.

The new dark theme is also available in the Proxmox VE API Viewer.

Local storage types that are located on other cluster nodes can be added.

A node selector was added to the Add Storage wizard for the ZFS, LVM, and LVM-Thin storage types.

Automatically redirect HTTP requests to HTTPS for convenience.

This avoids "Connection reset" browser errors that can be confusing, especially after setting up a Proxmox VE host the first time.

Task logs can now be downloaded directly as text files for further inspection.
It is now possible to choose the sort-order of the resource tree and to sort guests by name.
Fix loading of changelogs in case additional package repositories are configured.
Improve editing of backup jobs:
- Add a filter to the columns of the guest selector.
- Show selected, but non-existing, guests.
Remove the "Storage View" mode from the resource tree panel.

This mode only showed the storage of a cluster and did not provide additional information over the folder or server views.

The Proxmox Backup Server specific columns for verification and encryption status can now be used for sorting in the backup content view of a storage.
Polish the user experience of the backup schedule simulator by splitting the date and time into two columns and better check the validity of the input fields.
Improve accessibility for screens with our minimal required display resolution of 720p
- add scrolling overflow handler for the toolbar of the backup job view
- rework the layout of the backup job info window for better space usage and reduce its default size
Fix search in "Guests without backup" window.
Node and Datacenter resource summary panels now show the guest tag column by default.
Show role privileges when adding permissions.
Allow the use of the `-` character in snapshot names, as the backend has supported this for some time.
Update the noVNC guest viewer to upstream version 1.4.0.
Fix overly-strict permission check that prevented users with only the VM.Console privilege from accessing the noVNC console.
Align permission check for bulk actions with the ones enforced by the API.

Switch the check from the Sys.PowerMgmt privilege to the correct VM.PowerMgmt one.

Invalid entries in advanced fields now cause the advanced panel to unfold, providing direct feedback.
HTML-encode API results before rendering as additional hardening against XSS.
Fix preselection of tree elements based on the URL after login.
Fix race condition when switching between the content panel of two storage before one of them hasn't finished loading.
Metric server: Expose setting the verify-certificate option for InfluxDB as advanced setting
Replace non-clickable checkbox with icons for backup jobs, APT repositories, and replication jobs.
Fix error when editing LDAP sync setting and only a single parameter is not set to a non-default value.
Add missing online-help references for various panels and edit windows.
Improved translations, among others:
- Arabic
- French
- German
- Italian
- Japanese
- Russian
- Slovenian
- Simplified Chinese

Virtual Machines (KVM/QEMU)

New QEMU Version 7.2:
- QEMU 7.2 fixes issues with Windows Guests, installed from a German ISO, during installation of the VirtIO drivers.
- Fix crash of VMs with iSCSI disks on a busy target.
- Fix rare hang of VMs with IDE/SATA during disk-related operations like backup and resize.
- Many more changes, see the upstream changelog for details.
Taking a snapshot of a VM with large disks following a PBS backup occasionally was very slow. This has been fixed (issue #4476).
Running fsfreeze/fsthaw before starting a backup can now optionally be disabled in the QEMU guest agent options.

Note: Disabling this option can potentially lead to backups with inconsistent filesystems and should therefore only be disabled if you know what you are doing.

Cloning or moving a disk of an offline VM now also takes the configured bandwidth limits into consideration (issue #4249).
Fix an issue with EFI disks on ARM 64 VMs.
Add safeguards preventing the moving of disks of a VM using io_uring to storage types that have problems with io_uring in some kernel versions.
General improvements to error reporting. For example, the error messages from query-migrate are added when a migration fails and a configured, but non-existing physical CD-ROM drive, results in a descriptive error message.
Allow users to destroy a VM even if it's suspended.
Fix a race-condition when migrating VMs on highly loaded or slower clusters, where the move of the guest's config file to the target node directory might not have been propagated to the target node.
Rolling back a VM to a snapshot with state (memory) and still selecting to start the VM after the rollback does not cause an error anymore (rollbacks with state result in a running VM).
Deleting snapshots of running VMs, with a configured TPM on Ceph storages with krbd enabled, is now possible.
Fix command execution via pvesh and QEMU guest agent in VMs on other cluster nodes.
Update Linux OS version description to include 6.x kernels.

Containers (LXC)

Update to LXC 5.0.2 and lxcfs 5.0.3.
Allow riscv32 and riscv64 container architectures through the binfmt_misc kernel capability.

After installing the qemu-user-static and binfmt-support packages one can use a RISC-V based rootfs image to run as container directly on an x86_64/amd64 Proxmox VE host.

Create /etc/hostname file on Alma Linux, CentOS, and Rocky Linux containers. With this, DHCP requests sent by the container now include its hostname.
Add option to disconnect network interfaces of containers, similarly to network interfaces of VMs.
Make container start more resilient after OOM or node crash (empty AppArmor profile files do not cause a crash).
Improve cleanup upon failed restores (remove the container configuration if restore fails due to an invalid source archive, remove firewall configuration).
Ignore bind or read-only mount points when running pct fstrim.
During container shutdown, wait with a timeout in case lxc-stop fails. This prevents the shutdown task from running indefinitely and having to be aborted manually.
Templates:
- Updated Debian Bullseye template from 11.3 to 11.6.
- Updated Proxmox Mail Gateway template from 7.0 to 7.2.

General improvements for virtual guests

The "Bulk Stop" action was renamed to "Bulk Shutdown" to better describe its behavior.
Allow overriding timeout and force-stop settings for bulk shutdowns.
Allow bulk actions even if the user does not have the required privileges for all guests but has the privileges for each guest involved in the bulk action.

HA Manager

Add CRM command to switch an online node manually into maintenance (without reboot).

When a node goes into maintenance mode all active HA services will be moved to other nodes, but automatically migrate them back once the maintenance mode is disabled again.

The HA Cluster Resource Scheduler (CRS) stack was expanded to rebalance VMs & CTs automatically on start, not only recovery.

One can now enable the ha-rebalance-on-start option in the datacenter.cfg or via the web UI to use Proxmox CRS to balance on service start up.

A new intermediate state request_started has been added for the stop -> start transitions of services.
Improve scheduling algorithm for some cases.
- make CPU load matter more if there is no memory load at all
avoids boosting tiny relative differences over higher absolute loads.
- use a non-linear averaging algorithm when comparing loads.
The previous algorithm was blind in cases where the static node stats are the same and there is (at least) one node that is over committed when compared to the others.

Improved management for Proxmox VE clusters

Ensure that the current working directory is not in /etc/pve when you set up the cluster using the pvecm CLI tool.

Since pmxcfs, which provides the mount point for /etc/pve, is restarted when you set up the cluster, a confusing "Transport endpoint is not connected" error message would be reported otherwise.

The proxmox-offline-mirror tool now supports fetching data through an HTTP proxy.
Fetching the changelog of package updates has been improved:
- The correct changelog will be downloaded if repositories from multiple Proxmox projects are configured, for example if one has Proxmox VE and Proxmox Backup Server installed on the same host.
- Support getting the for packages coming from a Debian Backports repository.
You can now configure if you want to receive a notification mail for new available package updates.
The wrapper for acme.sh DNS-validation plugins received fixes for 2 small issues:
- a renaming of parameters for the acmedns plugin was pulled from upstream.
- a missing method was added to fix an issue with the dns_cf.sh plugin.
Improved pvereport: In order to provide a better status overview, add the following information:
- /etc/pve/datacenter.cfg.
- ceph health detail.
OpenSSL errors are now reported in full to ease troubleshooting when managing the nodes certificate.
Add missing or newly added/split-out packages to the Proxmox VE apt version API, also used for the pveversion -v call:
- proxmox-mail-forward
- proxmox-kernel-helper
- libpve-rs-perl

Backup/Restore

Suppress harmless but confusing "storing login ticket failed" errors when backing up to Proxmox Backup Server.

Storage

It is now possible to override the specific subdirectories for content (ISOs, container templates, backups, guest disks) to custom values through the content-dirs option.
The CIFS storage type can now also directly mount a specific subdirectory of a share, thus better integrating into already existing environments.
The availability check for the NFSv4 storage type was reworked in order to work with setups running without rpcbind.
Fix ISO upload via HTTP in a few edge cases (newlines in filenames, additional headers, not sent by common browsers).
Fix caching volume information for systems which both have a local ZFS pool storage and a ZFS over iSCSI storage configured during guest disk rescan.

Storage Replication

Extend support for online migration of replicated VM guests.

One can now also migrate VMs if they included snapshots, as long as those are only on replicated volumes.

Disk Management

Improve showing the SMART values for the correct NVMe devices.

Ceph

Expose more detailed OSD information through the API and use that to add an OSD Detail window in the web interface.

You can now check the backing device, logical volume info, front- and back- network addresses and more using the new OSD detail window.

Show placement groups per OSD in the web interface.
Improve schema description for various Ceph-related API endpoints.

This also improves the api-viewer and pvesh tool for various Ceph-related API endpoints.

Fix broken cmd-safety endpoint that made it impossible for non-root users to stop/destroy OSDs and monitors.
Allow admins to easily set up multiple MDS per node to increase redundancy if more than one CephFS is configured.

Access Control

ACL computation was refactored causing a significant performance improvement (up to a factor of 450) on setups with thousands of entries.
It is now possible to override the remove-vanished settings for a realm when actively syncing it in the GUI.
Allow quoted values in LDAP DN attributes when setting up an LDAP realm.

Firewall & Software Defined Networking

ipsets can be added even with set host-bits. For example, 192.0.2.5/24 is now a valid input. Host-bits get cleared upon parsing (resulting in 192.0.2.0/24 in the example).
Firewall logs can be restricted to a timeframe with the since and until parameters to the API call
The conditional loading of nf_conntrack_helpers was dropped for compatibility with kernel 6.1.
Not adding link-local IPv6 addresses on the internal guest-communication devices was fixed in a corner-case.
The MTU is now set to the value of the parent bridge on the automatically generated VLAN-bridge devices for non-VLAN-aware bridges.
The EVPN plugin now also merges a defined prefix-list from /etc/frr/frr.conf.local.

Installation ISO

the version of BusyBox shipped with the ISO was updated to version 1.36.0.
The EFI System Partition (ESP) defaults to 1 GiB of size if the root disk partition (hdsize) is bigger than 100 GB.
UTC can now be selected as timezone during installation.

Notable bug fixes

An issue with OVS network configuration where the node would lose connectivity when upgrading Open vSwitch (see https://bugs.debian.org/1008684).
A race condition in the API servers causing failed tasks when running a lot of concurrent API requests was fixed.

Known Issues & Breaking Changes

In QEMU 7.2, it is a hard error if audio initialization fails rather than a warning.

This can happen, for example, if you have an audio device with SPICE driver configured but are not using SPICE display. To avoid the issue, make sure the configuration is valid.

With pve-edk2-firmware >= 3.20221111-1 we know of two issues affecting specific set ups:
- virtual machines using OVMF/EFI with very little memory (< 1 GiB) and certain CPU types (e.g. host) might no longer boot.
Possible workarounds are to assign more memory or to use kvm64 as the CPU type.

The background for this problem is that OVMF << 3.20221111-1 used to guess the address (bit) width only from the available memory, but now there is more accurate detection that better matches what the configured CPU type provides. The more accurate address-width can lead to a larger space requirement for page tables.
- The (non-default) PVSCSI disk controller might result in SCSI disk not being detected inside the guess in regressions.
We're still investigating this, until then you might either evaluate if your VM really requires the non-standard PVSCSI controller, use the SATA bus instead, or keep using the older pve-edk2-firmware package.

Because of hardware errata, the XSAVES feature had to be disabled by the kernel for certain AMD CPUs, resulting in start issues with VMs and certain VM CPU types like EPYC-ROME.

For affected VMs you will find a message like host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3] logged in the kernel log.

As a workaround, use a different CPU model for the VM, like host, EPYC, or a custom CPU model that has the xsaves flag disabled.

Proxmox VE 7.3

Released 22. November 2022: See Downloads

Based on Debian Bullseye (11.5)
Latest 5.15 Kernel as stable default (5.15.74)
Newer 5.19 kernel as opt-in
QEMU 7.1
LXC 5.0.0
ZFS 2.1.6
Ceph Quincy 17.2.5
Ceph Pacific 16.2.10

Highlights

Ceph Quincy support. It is also the default for new installations
Initial Cluster Resource Scheduling (CRS) support
Tags for Virtual Guests in web-interface for better categorizing/searching/...
Support for Proxmox Offline Mirror to update and manage subscriptions of air gapped systems

Changelog Overview

Enhancements in the web interface (GUI)

Show tags for virtual guests in the resource tree and allow edits.
Improved UX for mediated PCIe devices - they now also show the name.
Improved Certificate View - for example for certificates with many SANs.
Node disk UI: gracefully handle adding the same local storage (e.g. a zpool with the same name) to multiple nodes.
Expose node configurations like wakeonlan and the delay for starting guests on boot for each node in the web interface.
Improved translations, among others:
- Arabic
- Dutch
- German
- Italian
- Polish
- Traditional Chinese
- Turkish
Improve rendering complex formats in the api-viewer widget

Virtual Machines (KVM/QEMU)

New major version of QEMU 7.1
Support for pinning a VM to certain CPU cores via taskset
In the web interface, new VMs default to iothread enabled and VirtIO SCSI-Single selected as SCSI controller (if supported by the guest OS)
New VMs use qemu-xhci USB controller, if supported by the guest OS (Windows >= 8, Linux >= 2.6)

USB devices can now be hot-plugged

Pass through up to 14 USB devices (previous 5) to a virtual machine

Align virtio-net parameters for the receive- (rx) and transmit- (tx) queue sizes with upstream best-practices
Use the more efficient packed format for multi-queues
Allow up to 64 rx and tx multi-queues (previously 16)
Cloud-init improvements: changes to the cloud-init settings now can be made available in the config-drive ISO inside the guest directly
Disable io_uring for CIFS backed VM disks by default - to mitigate an issue with CIFS and io_uring present since kernel 5.15
Improved handling for VMs with passed through PCIe-devices:
- Cleanup of created mdev devices, even if the VM could not be started
- Longer timeouts between sending SIGTERM and SIGKILL to allow for a cleanup upon termination
- Prevent suspending a VM with passed through PCIe device, as the device's state cannot be saved

Containers (LXC)

New major LXC version 5.0.0
More robust cgroup mode detection, by explicitly checking the type of /sys/fs/cgroup
Support for newer distribution versions:
- Fedora 37 and preparation for 38
- Devuan 12 Daedalus
- Preparation for Ubuntu 23.04
Bind-mounts are now also directly applied to a running container
Fix a bug when cloning a locked container: It does not create an empty config anymore, but fails correctly
Improvements to the systemd version detection inside containers
Volumes are now always deactivated upon successful move_volume, not only if the source volume is to be removed: preventing dangling krbd mappings
New pre-made templates available for:
- AlmaLinux 9
- Alpine 3.16
- Centos 9 Stream
- Fedora 36
- Fedora 37
- OpenSUSE 15.4
- Rocky Linux 9
- Ubuntu 22.10
Refreshed existing templates:
- Gentoo (2022-06-22-openrc)
- ArchLinux (2022-11-11)

General improvements for virtual guests

Add option to disable MAC learning on bridges (the guest-mac addresses are added statically, no broadcast packets are flooded to those ports, thus no spurious answers are replied, which broke certain hoster network setups)
Improve cleanup of backup-jobs upon purging the configuration for a removed VM
Optionally restart a guest after rollback to snapshot
Framework for remote migration to cluster-external Proxmox VE hosts

HA Manager

- Cluster Resource Scheduling (CRS) tech-preview: Improve new-node selection for when the HA Manager needs to find a new host node for a HA service, in the following cases:
- recovering it after fencing its node
- on node shutdown, if the migrate shutdown-policy is enabled
- on HA group configuration changes, if the current node is not in the highest priority set anymore

Use the TOPSIS multi-criteria decision analysis method for finding a better target

Having established the CRS foundation, Proxmox developers plan to extend it with a dynamic load scheduler and live load balancing in future releases

Cluster

Fix a permission issue in the QDevice status API

The API call for obtaining the API status needs privileged access, but was run directly in the unprivileged daemon leading to spurious permission denied errors

Fix race-condition between writing corosync.conf and reloading corosync on update

Backup/Restore

Improved namespace support for the Proxmox Backup Server storage type
Improvements to the parsing of the template variables of the backup notes

The notes template for backups, introduced in Proxmox VE 7.2, received a number of bug-fixes and improvements

Added option repeat-missed, allowing one to opt-out from the default behavior of running missed jobs on a new boot
The VM used for single-file-restore with QEMU guests now has support for increasing its memory (e.g. to handle many ZFS datasets inside the guest)
Improved configuration validation with Proxmox Backup Server encryption (for example, do not fall back to plain-text if the encryption key is missing)
When deleting vzdump backup files the corresponding notes and log are also removed.

Storage

Support ZFS dRAID vdevs when creating a zpool via the API & GUI. dRAID improves recovery times when a disk failure occurs.

A dRAID setup makes most sense for either a large (15+) amount of disks, or a medium+ amount of huge disks (15+ TB).

Align SMART status API with Proxmox Backup Server fields
Support Notes and the Protected setting for backups stored on BTRFS storage types.

Storage Replication

Don't send mails on bogus errors: e.g. when a replication could not be started because the guest is currently being migrated.
Upon replication failure the first 3 retries are scheduled in a shorter time, before waiting for 30 minutes before retrying - improving the consistency upon short network hiccups.
Cleanup replication state of guests running on another note: as can happen after a HA-fence.
Make interaction of replication state and configuration changes more robust: e.g. in the case of first removing all volumes from one storage, and then removing the VM before the next replication was run.

pve-zsync

support --compressed option resulting in an already compressed dataset to be sent as is to the destination (thus removing the need to decompress and potentially re-compress on the target).

Ceph

Improved UX when creating new clusters

The network selection and duplicate IP checking was improved.

It's no longer possible to run into an error, by selecting a different node for the first monitor than the one you are connected to (prevents trying to create a monitor on nodes without installed Ceph packages).

Added heuristic checks if it is OK to stop or remove a ceph MON, MDS, or OSD service.

The Web UI will now show a warning if the removal / stop of a service will affect the operation of the cluster.

Support for installing Ceph Quincy via Proxmox VE CLI and GUI.

Access Control

Improve naming of WebAuthn parameters in the GUI.
Increase OpenID code size - compatibility with Azure AD as OpenID provider.
Only require write-access (quorum) to TFA config for recovery keys.

All other TFA methods only need read-access to the config. This makes it possible to login to a node, which is not in the quorate partition, even if your user has TFA configured.

Fix a hard to trigger update issue with rotating the private key used for signing the access tickets, resulting in falsely rejected API calls.
Fix creation of tokens for other users, by anyone except root@pam

a bug prevented user A from creating a token for user B, despite having the relevant permissions

Better logging for expired tokens.

Firewall, Networking & Software Defined Networking (tech-preview)

Fix setting MTU on setups using OVS.
ifupdown2 now handles point-to-point settings correctly
ifupdown2 can now add a OVSBrige with a vlan-tag as ports to an OVSBridge (fakebridge)
Fix updating MTU if a bridge-port is plugged into a different bridge.
Firewall security groups can now be renamed with the changes directly being picked up from pve-firewall
Stricter parsing of guest config files in pve-firewall, making it possible to actually disable the guest firewall while keeping the config file around.
Improved handling on externally added ebtables rules: If a rule was added to a table different than filter, pve-firewall still tried to parse and add it to the filter table upon rule compilation.

Improved management for Proxmox VE clusters:

Proxmox Offline Mirror: The tool supports subscriptions and repository mirrors for air-gapped clusters. Newly added proxmox-offline-mirror utility can now be used to keep Proxmox VE nodes, without access to the public internet up-to-date and running with a valid subscription.
New mail-forwarding binary proxmox-mail-forward: no functional change, but unifies the configuration for sending the system-generated mails to the email address configured for root@pam
Improvements to pvereport - providing a better overview of the status of a Proxmox VE node the following information was added/improved:
- ceph-device-list
- stable ordering of guests and network information
- proxmox-boot-tool output
- arcstat output

HTTP & REST-API Daemon

File-uploads now support filenames with spaces.
File-uploads now support files with a size smaller than 16 KB
Improved input sanitization of API URLs as additional security layer.

Installation ISO

Fixed the permissions of /tmp inside the installation environment (e.g. for the edge-case of users installing software manually from a debug-shell)
Make the size requirement of 8 GB a warning - most installations can run with less space, but might need adaptation after install (e.g. moving the log-destination to another device) - keep 2 GB as hard limit
Rework auto-sizing of root, guest-data and SWAP partitions & avoid creating the LVM-Thin pool in favor of root partition space on small setups.

Mobile App

update to flutter 3.0
support and target Android 13
fix buttons hidden behind Android's soft nav buttons
provide feedback about running backup tasks: a bug prevented any visual feedback in the app, when starting a backup (although the backup was started).

Known Issues & Breaking Changes

Virtual Guest Tags:
Duplicate tags are now filtered out when updating the tag property of a virtual guest.

Duplication detection and sorting is handled case-insensitive by default, all tags are treated lower case. One can change that in the datacenter.cfg configuration using the case-sensitive boolean property of the tag-style option.

Proxmox VE 7.2

Released 4. May 2022: See Downloads

Based on Debian Bullseye (11.3)
Ceph Pacific 16.2.7
Ceph Octopus 15.2.16 (continued support until mid 2022)
Kernel 5.15.30
LXC 4.0
QEMU 6.2
ZFS 2.1.4

Changelog Overview

Enhancements in the web interface (GUI):
- Enhance backup restore dialog by allowing administrators to override some basic settings for the virtual guest being restored.
- Allow for safe reassignment of a VM disk or CT volume to another guest on the same node.
- Support configuring the range that new VMIDs are selected from when creating a VM or CT.
You can set the upper and lower boundaries in the datacenter's options panel. Setting lower equal to upper disables auto-suggestion completely.
- Enable passing the keyring secret for external Ceph clusters in the web interface when adding an RBD or CephFS storage to Proxmox VE.
- Render "ghost" OSDs in the Ceph dashboard to make admins aware of such leftover entries.
- Provide the "next-run" time, i.e., the time a backup job will be executed next, in the datacenter's backup job overview.
- Display a prominent Start Guest button in the Console view of guests, if they are not running .
- Improve performance for the task log viewer in the GUI.
- Move the actions affecting a disk or volume of a virtual guest to its own submenu.
- Switch some bitmap-based VM & CT resource icons over to vector based SVG.
- Allow assigning up to 16 PCI(e) devices to a VM via the web interface.
The previous limit of 5 has already expanded in the backend.
- Improved translations, among others:
  - Arabic
  - Basque
  - German
  - Japanese
  - Polish
  - Traditional Chinese
  - Turkish

Virtual Machines (KVM/QEMU)
- Add support for the accelerated virtio-gl (VirGL) display driver.
- Enable SPICE support by default for the VirtIO and VirGL display types.
- If VMs were paused before migration starts, keep them paused after migration finishes.
- Support for importing disk-images via the API - currently not available in the GUI.
- Support for reading OVF files via the API - currently not available in the GUI.
- The PCI vendor and device IDs can be overridden for a configuration passthrough entry, and do not need to correspond to the actual IDs of the physical device.
- Increase the timeout for the fsthaw operation issued through the QEMU guest agent.
- Fixed issues that occur in certain storage layouts, when migrating local VM state volumes, such as the TPM state or cloud-init.
- Improved clean-up of unneeded resources upon restore, for example, if a restore fails, drop the left-over snapshots of replaced disks and cloud-init disks.
- Improved consistency when cloning from a snapshot (rather than from the current state) - the EFI disk's state is also taken from the snapshot.

Containers (LXC)
- Support for NixOS container templates
- Support Ubuntu 22.04, Devuan 4.0 and the upcoming Fedora 36
- Provide new pre-made templates for:
  - Ubuntu 22.04
  - Devuan 4.0
  - Alpine 3.15
- Refreshed some existing templates:
  - CentOS 8 Stream,
  - Arch Linux
  - Debian 11 (to the 11.3 point release)
- Containers with multiple mount points can now be successfully trimmed with pct fstrim, even if one of the mount points does not support trimming, for example, because it's on a ZFS subvolume.
- Add target-storage support to migration.
- Allow lazytime mount option for containers.
- Fix an issue with wrong ownership for files/directories created by the container stack on unprivileged containers.
Files on Red Hat and SUSE based templates were created as root on the host and showed up as owned by nobody inside the container, resulting in NetworkManager failing.

General improvements for virtual guests
- Better handling of snapshot removal (for example, after finishing a backup) when storage replication is configured.

HA Manager
- Improve handling of huge numbers of services per node, by improving the LRM scheduler that starts workers.
ha-manager uses a statically configurable number of workers to handle the services, by scheduling jobs in repeating scheduling rounds with a time-slice of 10s max.

Since the services that have already started must also be checked to ensure that they are still in the target state, it could happen during large deployments that the services starved at the end of the queue.

The issue was fixed by additionally sorting the services by the amount of time in which they hadn't been scheduled. This increased the number of configurable services to be well above the previous implementation.
- Handle an edge case where a node would get stuck in fence state, if all services were removed from it before the node actually fenced itself.
- The ha-simulator can now better help to test races in scheduling (on the different nodes) by introducing a skip-round

Cluster
- A segfault triggered with certain usage patterns was fixed.
The issue was reported in the context of a cluster having many sessions opened with username and password (instead of API tokens) by a monitoring system.
- The lower and upper boundaries to be considered when selecting the next free VMID can now be set in the datacenter.cfg.
The settings are available over the web interface's Datacenter -> Options panel.

Backup/Restore
- Allow for the configuration of a notes-template for backup jobs, which will be evaluated once the job runs and added to each resulting backup of said job.
With this, useful meta-information can be added, allowing backups to be easier to distinguish, so that a relevant one can be found quickly if required.

Note templates can contain template variables like {{guestname}} or {{cluster}}.
- The next-event scheduling code was updated and now uses the more powerful one from Proxmox Backup Server via a Perl to rust binding (perlmod).
In addition to the existing weekday, time and time range support, you can now specify:
- - Dates: For example 2015-10-21 or *-12-31 23:50 (New Year's Eve, 10 minutes before midnight every year)
  - Date ranges: Sat *-1..7 15:00 (first Saturday every month at 15:00)
  - Repeating ranges Sat *-1..7 */30 (first Saturday every month, every half hour)
- Add a 'max-protected-backups' property to the storage definitions, to address issues with running into the configured general limit, due to having protected backups on the storage which should not count towards it.
- Add a job-init hook that is run once upfront, on each node that hosts guests which are included in the backup job.

Storage
- Increase the timeout of ZFS operations for synchronous and worker calls to 10 seconds and a minimum of 5 minutes respectively, based on a report of the current settings being too low.

pve-zsync
- Improved resiliency of the crontab parser to better cope with manual modifications.
- Support syncing TPM state.

Ceph
- Support creating and destroying erasure coded pools, which can also be added as Proxmox VE Storage entries.
- Support for selecting existing custom device classes when adding OSDs through the web interface.
- Out/In commands for OSDs are now executed on the node the user is connected to, instead of the node where the OSDs are located.
This makes it possible to disable OSDs in the GUI, in case the target node is not available (for example in a disaster recovery situation).

Access Control
- More fine-grained control over how users or groups that vanished while syncing realms should be handled.
Administrators can now choose if the user and all their configured values (for example TFA configurations, ACL entries) are set to the state in LDAP, or if only the users are added/removed, while their properties remain.

This makes it possible to handle removed users, without losing the TFA configuration for all users who did not change since the last sync.
OpenID realms support configuring of scopes, authentication context class reference (ACR) values, and whether the provider prompts the user for re-authentication and consent.
- Fixed an issue with using Azure as an OpenID provider (unexpected Content-Type-Encoding leading to rejected logins).
- WebAuthn TFA configurations now support only optionally providing the origin parameter for clustered systems.
- Improved robustness in case of misconfigured U2F and WebAuthn settings.
Logins with other TFA methods do not get rejected if the U2F/WebAuthn configuration is incorrect.
- TFA settings can now be viewed by users with User.Modify and Sys.Audit permissions in addition to root@pam

Firewall & Software Defined Networking (tech preview)
- Set the ICMP type in the firewall rules GUI directly.
- Add support for configuring anycast addresses on guests, by advertising the whole subnet to the EVPN network.
- Add configuration for allowing exit-notes to reach guests inside the EVPN network, for example if they need to talk with a DNS server running inside a virtual guest.
- Support for floating IPs on guests (by not suppressing ARP/ND packets).
- Hide bridges that a user doesn't have access to, if vNets are in use.

Improvements for managing a Proxmox VE system
- It is now possible to select a particular kernel version to boot persistently from a running system through proxmox-boot-tool kernel pin
You can select a particular version to be used either indefinitely or just for the next boot. This removes the need to watch the boot process to select the desired version in the bootloader screen.
- Administrators can now disable MAC learning on a bridge in /etc/network/interfaces with the bridge-disable-mac-learning flag.
This reduces the number of packets flooded on all ports (for unknown MAC addresses), preventing issues with certain hosting providers (for example, Hetzner), which resulted in the Proxmox VE node getting disconnected.
- The location of the private key-file used for pveproxy can now be set in /etc/default/pveproxy, supporting certain setups which need sensitive data to be stored in a different location due to regulatory requirements.
- The TLS configuration options for pveproxy now support disabling TLS 1.2 (to run in a pure TLS 1.3 environment) and configuring the advertised TLS1.3 ciphers.
- Client-initiated TLS renegotiation was disabled, following best-practices.
- The pveschedule daemon was improved based on the feedback received from its initial release with Proxmox VE 7.1.
- Data sent to a configured metrics server is now formatted in a more strict fashion to prevent type mismatches, which were possible if a guest had a numeric-only name.

Mobile App
- Provide a simple dark theme and enable it if the system settings are configured to use dark designs.
- Provide an inline console by relaying noVNC for VMs, and xterm.js for containers and the Proxmox VE node shell in the GUI.
- Improved support for notches and full-screen mode.

Installation ISO
- Updated memtest86+ to the completely rewritten 6.0b version, adding support for UEFI and modern memory like DDR5.
- Support for restricting the partitioned space for BTRFS installs, in order to leave unpartitioned space, which can be used for swap, for example.
- Improved accuracy of displayed disk size by adding a decimal point. Otherwise, a 1.8 TB disk, for example, was shown as being just 1 TB.
- Support for creating RAID0 configurations with differently sized disks.
- Improved UX when setting the hdsize for RAID setups.
- ZFS installs can now be configured to use zstd or GZIP compression.
- Better support for installing ZFS on Systems with certain bugs in their UEFI implementation (HP Thin clients).
- Drop the kernel command line quiet flag when booting in the advanced mode to provide more verbosity.

Notable bug fixes
- To prevent a network outage during the transition from ifupdown to ifupdown2, the ifupdown package was modified to not stop networking upon its removal.
- For backups of running VMs with multiple disks, it was possible that write operations happening after the backup start synchronization point to the non-primary disks to be included in the backup.
This has been fixed with QEMU 6.2.0 by initializing the relevant bitmap early enough.

Known Issues

Setups mounting a QNAP NFS share could not be mounted with NFS version 4.1 with kernel pve-kernel-5.15.30-2-pve - the issue has been mitigated in kernels pve-kernel-5.15.35-2 and above.
If your QNAP NFS share cannot be mounted upgrade the kernel and reboot.

As an alternative mitigation you can explicitly set the NFS version to 4 (not 4.1 or auto).
PCI(e) pass through related:
- Systems passing through a GPU may be affected from the switch to the SYS_FB (system frame buffer) KConfig build options using the simplefb module as driver in the new default 5.15 based kernel.
  The sys-fb allows taking over the FB from the firmware/earlier boot stages. Note that Proxmox VE uses the legacy simplefb driver over the modern simpledrm one due to regressions and issues we encountered on testing with the latter.
  
  Most of those issues are already fixed in newer kernels and Proxmox VE may try to switch to the modern, DRM based FB driver once it moves to 5.17, or newer, as its default kernel.
  
  If your systems is configured to pass through the (i)GPU, and you had to avoid the host kernel claiming the device, you may now need to also add video=simplefb:off to the kernel boot command line.
- Setups using vendor-reset for PCIe pass through need to adapt to changes of the new default 5.15 based kernel, see For details see this issue.
  They must run the command echo 'device_specific' > /sys/bus/pci/devices/<PCI-ID>/reset_method before the VM is started. This can be automated by using a systemd service or using a on-boot cron script.
  
  Alternatively one can also use a VM hook script with the pre-start hook.
intel_iommu defaulted to on in kernels 5.15 based kernels up to pve-kernel-5.15.35-3-pve with version 5.15.35-6. An enabled IOMMU can cause problems with older hardware - issues were reported with, for example, HP DL380 g8 servers, and Dell R610 servers (both older than 10 years).
The issue can be fixed by explicitly disabling intel_iommu on the kernel commandline (intel_iommu=off) following the reference documentation - https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline

As the setting has been reverted in newer pve-kernel-5.15 packages the issue is now mostly relevant during installation from the ISO.
Certain systems may need to explicitly enable iommu=pt (SR-IOV pass-through) on the kernel command line.
There are some reports for this to solve issues with Avago/LSI RAID controllers, for example in a Dell R340 Server booted in legacy mode.
~~Certain systems may need to work around a HW issue with TDP (two-dimensional paging), a now by default enabled technic to speed up VM to host, or vice versa, switches.~~
~~If you see a kernel log message containing KVM: entry failed, hardware error 0x80000021 you are affected.~~

This has been adressed with the kernel package pve-kernel-5.15.39-3-pve with version 5.15.39-3 and newer, which is available on all repositories.

Proxmox VE 7.1

Released 17. November 2021: See Downloads (updated ISO release 2 with updated kernel and bug fixes 7. December 2021)

Based on Debian Bullseye (11.1)
Ceph Pacific 16.2.6
Ceph Octopus 15.2.15 (continued support until mid 2022)
Kernel 5.13 default
LXC 4.0
QEMU 6.1
ZFS 2.1

Changelog Overview

Enhancements in the web interface (GUI):
- Improved, more flexible scheduling options for backup jobs.
Backup jobs are now run by a new scheduler daemon (pvescheduler) and support the same more flexible calendar-event schedules already known from the verify and garbage-collection jobs in Proxmox Backup Server.
- Support configuration of backup retention policy for each backup job.
Brings the powerful and flexible retention settings from the API and CLI to the GUI, making it easier to set up smart backup policies.
- Add Schedule Simulator tool to the backup job overview panel to help get the calendar event format right.
- Simplify the creation of virtual guests with multiple disks.
Additional disks can now be added from the creation wizard, eliminating the need to add them after creating the VM or Container.
- Added a combo box under advanced options, for selecting the asynchronous IO mode for each disk of a virtual machine.
- Add a Notes panel for the datacenter. It can interpret Markdown and render it as HTML.
- Improvements to the UX for configuring metric servers.
- Two-factor authentication improvements:
  - Support for WebAuthn (superseding U2F) and one-time recovery keys.
  - Support for configuring multiple factors for a single account.
- Improved translations, among others:
  - Arabic
  - Basque
  - Brazilian Portuguese
  - French
  - German
  - Simplified Chinese
  - Traditional Chinese
  - Turkish

Virtual Machines (KVM/QEMU):
- Support for Windows 11 OS type
- UEFI Secure Boot
Newly created VMs have support for secure boot verification, with an option to have Linux distribution and Microsoft keys pre-enrolled.
- TPM support
Support for adding a Trusted Platform Module to any VM, fulfilling the requirements of modern operating systems, for example Windows 11.
- Store the current time and the installed QEMU version, when creating a VM.
This information is also used to work around changes in the QEMU PCI/ACPI device hotplug layer, which might cause network interface name-changes otherwise.
- SCSI and VirtIO VM disks can be marked read-only.
- Allow re-assigning a disk to another VM.
- qm move_disk has been renamed to qm move-disk.
The old syntax is still supported for backwards compatibility.
- Changed the initial default resolution from 800x600 to 1024x768 when using OVMF (EFI).
Note that the resolution is saved in the EFIvars, so this mainly affects new VMs. You can override the preferred resolution in the OVMF settings menu (Device Manager -> Platform Config).

Container
- Support Fedora 35 and Ubuntu 21.10.
- Support and provide templates for the AlmaLinux and Rocky Linux distributions, which claim to be 1:1 binary compatible with RHEL.
- Support zstd-compressed templates.
Leverage the highly efficient, high-ratio, state-of-the-art compression algorithm for CT templates.
- Enable the nesting feature for new unprivileged containers created via the web interface.
  The nesting feature ensures better interoperability with modern systemd versions used in newer templates.
- Better handling of unmanaged containers.
- Allow re-assigning a mount point volume to another container.
- pct move_volume has been renamed to pct move-volume.
The old syntax is still supported for backwards compatibility.
- Improved support for restoring backups of privileged containers as unprivileged ones (use of unprivileged containers is strongly recommended wherever possible).
- Added support for container CPU-sets on hosts with cgroupv2 layout.

General improvements for virtual guests
- Better interoperability between storage replication and rollback to an independent snapshot.

Cluster
- Updated corosync to include bug-fixes for issues occurring during network recovery.
This could have otherwise lead to loss of quorum on all cluster nodes, which in turn would cause a cluster-wide fencing event in case HA was enabled.

Disk Management
- Support wiping specific partitions.
Added more fine-grained control over what part of a disk you want to clean up for re-use.
- Support creating storages on a particular partition, in addition to using the whole disk.
- Support using partitions for specific use-cases, like Ceph Journal.
- Support for zstd compression when creating a new zpool via API/GUI.

Storage
- Make pre-allocation policy configurable for file-based storage.
Fully pre-allocated images can provide better performance, but take a longer time when being created - see qemu-img(1) for details.

Backup/Restore
- Augmented scheduling options for backups.
Backup jobs are now run by a new scheduler daemon (pvescheduler) and support the more flexible schedule-options known from Proxmox Backup Server verify and garbage collection jobs (which in turn are inspired by systemd-timers).
- Support for protected backups, which will not be pruned and cannot be removed manually, without removing the protected flag.

Storage Replication
- Scheduling of replication jobs is now handled by the pvescheduler daemon instead of a systemd-timer running every minute, resulting in vastly reduced logging on the nodes.

Ceph
- Full support for configuring external Ceph clusters via API.
Before, the keyring needed to be put in the correct place via shell.
- Multiple CephFS instances are supported.

Access Control
- OpenID Authentication now honors the configured http-proxy in the datacenter settings.
- Two-Factor Authentication
  - Add support for multiple 2nd factors.
  - Add recovery keys as 2nd factor.
  - Add WebAuthn 2nd factor support.
  - Rework TFA and login UI accordingly.

Improvements regarding the upgrade from version 6.4:
- A bug in proxmox-boot-tool, which could make a Proxmox VE system running the 5.11 kernel unbootable upon upgrade, was fixed.
- Privileged containers with old systemd versions can now run successfully - the Proxmox VE node still needs to be booted into the legacy cgroup layout.

Noteable or Breaking Changes

API deprecations, moves and removals
- The /version API method now only returns the console setting of the datacenter config, instead of all settings (use the /cluster/options instead).

CLI changes
- qm: rename move_disk command to move-disk with an alias for backward compatibility.

Config file changes
- The /etc/sysctl.d/pve.conf snippet has been moved to /usr/lib/sysctl.d/10-pve.conf, in accordance with current recommendations (see sysctl.d(5)). Local modifications need to be re-enabled again - they are preserved in /etc/sysctl.d/pve.conf.dpkg-old).

Two-Factor Authentication:
- Keys stored by admins in the user entry (this is how it worked back before the users could add TOTP and U2F keys themselves, and how it worked for Yubico OTP until now) are now deprecated.
- The two-step login procedure in the API has been changed to work like in the Proxmox Backup Server. For now, the new procedure is opt-in and the old one still works, but is deprecated.

Known Issues

Web-interface glitches
- Please reload the web-interface after the upgrade to avoid various glitches or issues all around.
Ensure that the browser caches are circumvented on reload, for example by using CTRL + SHIFT + R (switch CTRL with CMD for MacOS).

Two-Factor Authentication:
- Keys stored by admins in the user entry (see above) still work, but will not be visible in the TFA panel until that user’s TFA configuration has been updated once, by registering any new TFA entry, such as by adding recovery keys.
- When in a cluster, changes to TFA entries can only be done after all nodes have been upgraded, since the TFA configuration file format changed and will be automatically upgraded to the new format with the first change.
- Old API clients which support 2nd factors via two-step logins may be unable to log in to accounts which have multiple TFA entries, or ones which were not supported previously.

With the addition of pvescheduler - the new backup schedules are no longer configured in the cluster-wide /etc/pve/vzdump.cron file. While this file is still read and will remain supported for the foreseeable future, changing an existing backup schedule will cause the schedule to be stored in the new config file /etc/pve/jobs.cfg.
This is not problematic, as long as not all nodes in a cluster have been upgraded to the new version, since backups will only run on nodes who read the new config file.

To avoid any problems, simply upgrade all nodes in your cluster before making any changes to scheduled backups.

Low potential for full-cluster fencing event when upgrading HA-enabled clusters using libknet1 version 1.21-pve1
- Under certain network load situations, that is one set of nodes having very low left-over bandwidth and another set has a lot of available bandwidth, a cluster (partly) running libknet1 version 1.21-pve1 can trigger a bug in corosync/kronosnet. With HA enabled this can then lead to a fencing event for the entire cluster.
If a separate network is used for corosync/kronosnet the chance that the bug can get triggered is highly reduced.

See bug #3672 for details.
- As a precaution, it is advised to disable HA, in order to prevent a fencing event. This is done by stopping the `pve-ha-lrm` service on all nodes, then stopping the `pve-ha-crm` service on all nodes prior to upgrading. After all nodes have been upgraded, HA should be enabled again before rebooting any nodes, by first restarting `pve-ha-lrm` on all nodes, then restarting `pve-ha-crm` on all nodes.

Garbled output during installation of Debian in VMs when standard VGA display and OVMF/UEFI is configured
- Debian's Kernel package currently does not enable the simplefb module and thus uses the legacy efifb frame buffer, this can lead to glitches with the OVMF version that Proxmox VE 7.1 ship.
As workaround, you can either use QXL (SPICE) for the VM's display, use SeaBIOS or disable Secure Boot in OVMF and use the non-graphical, text based installer.

VMs with SATA as disk bus may show issues when used with io_uring
- For now this can be worked around by switching the disk's Async IO mode to threads (for cache = write back/through) or native (for cache = off, none or direct sync), respectively, in the Advanced options of the disk edit dialogue.
- Note that SATA is really not the best choice for a VM's disk bus in general, rather use (VirtIO-) SCSI for best performance and feature set. For Windows VirtIO support is available through the Windows VirtIO Drivers

Proxmox VE 7.0

Released 6. July 2021: See Downloads, updated ISO release 2 with updated base system to final Debian 11.0 Bullseye and bug fixes 31. August 2021

Based on Debian Bullseye (11)
Ceph Pacific 16.2 as new default
Ceph Octopus 15.2 continued support
Kernel 5.11 default
LXC 4.0
QEMU 6.0
ZFS 2.0.4

Changelog Overview

Installer:
- Rework the installer environment to use switch_root instead of chroot, when transitioning from initrd to the actual installer.
  This improves module and firmware loading, and slightly reduces memory usage during installation.
- Automatically detect HiDPI screens, and increase console font and GUI scaling accordingly. This improves UX for workstations with Proxmox VE (for example, for passthrough).
- Improve ISO detection:
  - Support ISOs backed by devices using USB Attached SCSI (UAS), which modern USB3 flash drives often do.
  - Linearly increase the delay of subsequent scans for a device with an ISO image, bringing the total check time from 20s to 45s. This allows for the detection of very slow devices, while continuing faster in general.
- Use zstd compression for the initrd image and the squashfs images.
- Setup Btrfs as root file system through the Proxmox VE Installer (Technology preview)
- Update to busybox 1.33.1 as the core-utils provider.

Enhancements in the web interface (GUI):
- The node summary panel shows a high level status overview, while the separate Repository panel shows in-depth status and list of all configured repositories. Basic repository management, for example, activating or deactivating a repository, is also supported.
- Notes panels for Guests and Nodes can now interpret Markdown and render it as HTML.
- On manually triggered backups, you can now enable pruning with the backup-retention parameters of the target storage, if configured.
- The storage overview now uses SI units (base 10) to be consistent with the units used in the graphs.
- Support for security keys (like YubiKey) as SSH keys, when creating containers or preparing cloud-init images.
- Improved rendering for IOMMU-groups when adding passthrough PCI devices to QEMU guests.
- Improved translations, among others:
  - Arabic
  - French
  - German
  - Japan
  - Polish
  - Turkish

Access Control:
- Single-Sign-On (SSO) with the new OpenID Connect access realm type.
You can integrate external authorization servers, either using existing public services or your own identity and access management solution, for example, Keycloack or LemonLDAP::NG.
- Added new permission Pool.Audit to allow users to see pools, without permitting them to change the pool.
See breaking changes below for some possible impact in custom created roles.

Virtual Machines (KVM/QEMU):
- QEMU 6.0 has support for io_uring as an asynchronous I/O engine for virtual drives - this is now the default for newly started or migrated guests.
The new default can be overridden in the guest config via qm set VMID --DRIVE EXISTING-DRIVE-OPTS,aio=native (where, for example, DRIVE would be scsi0 and the OPTS could be get from qm config VMID output).
- EFI disks stored on Ceph now use the writeback caching-mode, improving boot times in case of slower or highly-loaded Ceph storages.
- Unreferenced VM disks (not present in the configuration) are not destroyed automatically any more:
  - This was made opt-in in the GUI in Proxmox VE 6.4 and is now also opt-in in the API and with CLI tools.
  - Furthermore, if this clean-up option is enabled, only storages with content-types of VM or CT disk images, or rootdir will be scanned for unused disk-volumes.
With this new default value, data loss is also prevented by default. This is especially beneficial in cases of dangerous and unsupported configurations, for example, where one backing storage is added twice to a Proxmox VE cluster with an overlapping set of content-types.
- VM snapshot states are now always removed when a VM gets destroyed.
- Improved logging during live restore.

Container
- Support for containers on custom storages.
- Clone: Clear the cloned container's `/etc/machine-id` when systemd is in use or that file exists. This ID must be unique, in order to prevent issues such as MAC address duplication on Linux bridges.

Migration
- QEMU guests: The migration protocol for sending the Spice ticket changed in Proxmox VE 6.1. The code for backwards compatibility has now been dropped, prohibiting the migration of VMs from Proxmox VE 7.0 to hypervisors running Proxmox VE 6.1 and older.
Always upgrade to the latest Proxmox VE 6.4, before starting the upgrade to Proxmox VE 7.
- Containers: The force parameter to pct migrate, which enabled the migration of containers with bind mounts and device mounts, has been removed. Its functionality has been replaced by marking the respective mount-points as shared.

High Availability (HA):
- Release LRM locks and disable watchdog protection if all services of the node the LRM is running on, got removed and no new ones were added for over 10 minutes.
This reduced the possible subtle impact of an active watchdog after a node was cleared of HA services, for example, when HA services were previously only configured for evaluation.
- Add a new HA service state recovery and transform the fence state in a transition to that new state.
This gains a clear distinction between to be fenced services and the services whose node already got fenced and are now awaiting recovery.
- Continuously retry recovery, even if no suitable node was found.
This improves recovery for services in restricted HA groups, as only with that the possibility of a quorate and working partition but no available new node for a specific service exists.

For example, if HA is used for ensuring that a HA service using local resource, like a VM using local storage, will be restarted and up as long as the node is running.
- Allow manually disabling HA service that currently are in recovery state, for more admin control in those situations.

Backup and Restore
- Backups of QEMU guests now support encryption using a master key.
- It is now possible to back up VM templates with SATA and IDE disks.
- The maxfiles parameter has been deprecated in favor of the more flexible prune-options.
- vzdump now defaults to keeping all backups, instead of keeping only the latest one.
- Caching during live restore got reworked, reducing total restore time required and improving time to fully booted guest both significantly.
- Support file-restore for VMs using ZFS or LVM for one, or more, storages in the guest OS.

Network:
- Default to the modern ifupdown2 for new installations using the Proxmox VE official ISO. The legacy ifupdown is still supported in Proxmox VE 7, but may be deprecated in a future major release.

Time Synchronization:
- Due to the design limitations of systemd-timesync, which make it problematic for server use, new installations will install chrony as the default NTP daemon.
If you upgrade from a system using systemd-timesyncd, it's recommended that you manually install either chrony, ntp or openntpd.

Ceph Server
- Support for Ceph 16.2 Pacific
- Ceph monitors with multiple networks can now be created using the CLI, provided you have multiple public_networks defined.
Note that multiple public_networks are usually not needed, but in certain deployments, you might need to have monitors in different network segments.
- Improved support for IPv6 and mixed setups, when creating a Ceph monitor.
- Beginning with Ceph 16.2 Pacific, the balancer module is enabled by default for new clusters, leading to better distribution of placement groups among the OSDs.
- Newly created Bluestore OSDs will benefit from the newly enabled sharding configuration for rocksdb, which should lead to better caching of frequently read metadata and less space needed during compaction.

Storage
- Support for Btrfs as technology preview
  - Add an existing Btrfs file system as storage to Proxmox VE, using it for virtual machines, container, as backup target or to store and server ISO and container appliance images.
- The outdated, deprecated, internal DRBD Storage plugin has been removed. A derived version targeting newer DRBD is maintained by Linbit^{[footnotes 7.0 1]}.
- More use of content-type checks instead of checking a hard-coded storage-type list in various places.
- Support downloading ISO and Cont appliance images directly from a URL to a storage, including optional checksum verifications.

Disk Management
- Wiping disks is now possible from the GUI, enabling you to clear disks which were previously in use and create new storages on them. Note, wiping a disk is a destructive operation with data-loss potential.
Note that with using this feature any data on the disk will be destroyed permanently.

pve-zsync
- Separately configurable number of snapshots on source and destination, allowing you to keep a longer history on the destination, without the requirement to have the storage space available on the source.

Firewall
- The sysctl settings needed by pve-firewall are now set on every update to prevent disadvantageous interactions during other operations (for example package installations).

Certificate management
- The ACME standalone plugin has improved support for dual-stacked (IPv4 and IPv6) environments and no longer relies on the configured addresses to determine its listening interface.

Breaking Changes

Pool permissions

The old permission Pool.Allocate now only allows users to edit pools, not to see them. Therefore, Pool.Audit must be added to existing custom roles with the old Pool.Allocate to preserve the same behavior. All built-in roles are updated automatically.

VZDump
- Hookscript: The TARFILE environment variable was deprecated in Proxmox VE 6, in favor of TARGET. In Proxmox VE 7, it has been removed entirely and thus, it is not exported to the hookscript anymore.
- The size parameter of vzdump has been deprecated, and setting it is now an error.

API deprecations, moves and removals
- The upgrade parameter of the /nodes/{node}/(spiceshell|vncshell|termproxy) API method has been replaced by providing upgrade as cmd parameter.
- The /nodes/{node}/cpu API method has been moved to /nodes/{node}/capabilities/qemu/cpu
- The /nodes/{node}/ceph/disks API method has been replaced by /nodes/{node}/disks/list
- The /nodes/{node}/ceph/flags API method has been moved to /cluster/ceph/flags
- The db_size and wal_size parameters of the /nodes/{node}/ceph/osd API method have been renamed to db_dev_size and wal_dev_size respectively.
- The /nodes/<node>/scan/usb API method has been moved to /nodes/<node>/hardware/usb

CIFS credentials have been stored in the namespaced /etc/pve/priv/storage/<storage>.pw instead of /etc/pve/<storage>.cred since Proxmox VE 6.2 - existing credentials will get moved during the upgrade allowing you to drop fallback code.

The external storage plugin mechanism had a ABI-version bump that reset the ABI-age, thus marking an incompatible breaking change, that external plugins must adapt before being able to get loaded again.

qm|pct status <VMID> --verbose, and the respective status API call, only include the template line if the guest is a template, instead of outputting template: for guests which are not templates.

qm vncproxy now always requires a password to be set via the LC_PVE_TICKET environment variable.

Known Issues

Network: Due to the updated systemd version, and for most upgrades, the newer kernel version (5.4 to 5.11), some network interfaces might change upon reboot:
- Some may change their name. For example, due to newly supported functions, a change from enp33s0f0 to enp33s0f0np0 could occur.
  We observed such changes with high-speed Mellanox models.
- Bridge MAC address selection has changed in Debian Bullseye - it is now generated based on the interface name and the machine-id (5) of the system.
Systems installed using the Proxmox VE 4.0 to 5.4 ISO may have a non-unique machine-id. These systems will have their machine-id re-generated automatically on upgrade, to avoid a potentially duplicated bridge MAC.

If you do the upgrade remotely, make sure you have a backup method of connecting to the host (for example, IPMI/iKVM, tiny-pilot, another network accessible by a cluster node, or physical access), in case the network used for SSH access becomes unreachable, due to the network failing to come up after a reboot.

Container:
- cgroupv2 support by the container’s OS is needed to run in a pure cgroupv2 environment. Containers running systemd version 231 or newer support cgroupv2 [1], as do containers that do not use systemd as init system in the first place (e.g., Alpine Linux or Devuan).
CentOS 7 and Ubuntu 16.10 are two prominent examples for Linux distributions releases, which have a systemd version that is too old to run in a cgroupv2 environment, for details and possible fixes see: https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup_compat

Proxmox VE 7.0 beta1

Released 24. June 2021: See Downloads

Based on Debian Bullseye (11)
Ceph Pacific 16.2 as new default
Ceph Octopus 15.2 continued support
Kernel 5.11 default
LXC 4.0
QEMU 6.0
ZFS 2.0.4
Installer:
- Rework the installer environment to use switch_root instead of chroot, when transitioning from initrd to the actual installer.
  This improves module and firmware loading, and slightly reduces memory usage during installation.
- Automatically detect HiDPI screens, and increase console font and GUI scaling accordingly. This improves UX for workstations with Proxmox VE (for example, for passthrough).
- Improve ISO detection:
  - Support ISOs backed by devices using USB Attached SCSI (UAS), which modern USB3 flash drives often do.
  - Linearly increase the delay of subsequent scans for a device with an ISO image, bringing the total check time from 20s to 45s. This allows for the detection of very slow devices, while continuing faster in general.
- Use zstd compression for the initrd image and the squashfs images.
- Update to busybox 1.33.1 as the core-utils provider.

Enhancements in the web interface (GUI)
- Notes panels for Guests and Nodes can now interpret Markdown and render it as HTML.
- On manually triggered backups, you can now enable pruning with the backup-retention parameters of the target storage, if configured.
- The storage overview now uses SI units (base 10) to be consistent with the units used in the graphs.
- Support for security keys (like YubiKey) as SSH keys, when creating containers or preparing cloud-init images.
- Improved rendering for IOMMU-groups when adding passthrough PCI devices to QEMU guests.
- Improved translations, among others:
  - Arabic
  - French
  - German
  - Polish

Virtual Machines (KVM/QEMU):
- QEMU 6.0 has support for io_uring as an asynchronous I/O engine for virtual drives - this is now the default for newly started or migrated guests.
The new default can be overridden in the guest config per drive via qm set VMID --DRIVE EXISTING-DRIVE-OPTS,aio=native.
- EFI disks stored on Ceph now use the writeback caching-mode, improving boot times in case of slower or highly-loaded Ceph storages.
- Unreferenced VM disks (not present in the configuration) are not destroyed automatically any more:
  - This was made opt-in in the GUI in Proxmox VE 6.4 and is now also opt-in in the API and with CLI tools.
  - Furthermore, if this clean-up option is enabled, only storages with content-types of VM or CT disk images, or rootdir will be scanned for unused disk-volumes.
With this new default value, data loss is also prevented by default. This is especially beneficial in cases of dangerous and unsupported configurations, for example, where one backing storage is added twice to a Proxmox VE cluster with an overlapping set of content-types.
- VM snapshot states are now always removed when a VM gets destroyed.
- Improved logging during live restore.

Container
- Support for containers on custom storages.
- Clone: Clear the cloned container's `/etc/machine-id` when systemd is in use or that file exists. This ID must be unique, in order to prevent issues such as MAC address duplication on Linux bridges.

Migration
- QEMU guests: The migration protocol for sending the Spice ticket changed in Proxmox VE 6.1. The code for backwards compatibility has now been dropped, prohibiting the migration of VMs from Proxmox VE 7.0 to hypervisors running Proxmox VE 6.1 and older.
Always upgrade to the latest Proxmox VE 6.4, before starting the upgrade to Proxmox VE 7.
- Containers: The force parameter to pct migrate, which enabled the migration of containers with bind mounts and device mounts, has been removed. Its functionality has been replaced by marking the respective mount-points as shared.

Backup and Restore
- Backups of QEMU guests now support encryption using a master key.
- It is now possible to back up VM templates with SATA and IDE disks.
- The maxfiles parameter has been deprecated in favor of the more flexible prune-options.
- vzdump now defaults to keeping all backups, instead of keeping only the latest one.

Network:
- Default to the modern ifupdown2 for new installations using the Proxmox VE official ISO. The legacy ifupdown is still supported in Proxmox VE 7, but may be dropped in a future major release.

Time Synchronization:
- Due to the design limitations of systemd-timesync, which make it problematic for server use, new installations will install chrony as the default NTP daemon.
If you upgrade from a system using systemd-timesyncd, it's recommended that you manually install either chrony, ntp or openntpd.

Ceph Server
- Support for Ceph 16.2 Pacific
- Ceph monitors with multiple networks can now be created using the CLI, provided you have multiple public_networks defined.
Note that multiple public_networks are usually not needed, but in certain deployments, you might need to have monitors in different network segments.
- Improved support for IPv6 and mixed setups, when creating a Ceph monitor
- Beginning with Ceph 16.2 Pacific, the balancer module is enabled by default for new clusters, leading to better distribution of placement groups among the OSDs
- Newly created Bluestore OSDs will benefit from the newly enabled sharding configuration for rocksdb, which should lead to better caching of frequently read metadata and less space needed during compaction

Storage
- The outdated, deprecated, internal DRBD Storage plugin has been removed. A derived version targeting newer DRBD is maintained by Linbit^{[footnotes 7.0 1]}.

Disk Management
- Wiping disks is now possible from the GUI, enabling you to clear disks which were previously in use and create new storages on them. Note, wiping a disk is a destructive operation with data-loss potential. Any data on the disk will be destroyed.

pve-zsync
- Separately configurable number of snapshots on source and destination, allowing you to keep a longer history on the destination, without the requirement to have the storage space available on the source.

Firewall
- The sysctl settings needed by pve-firewall are now set on every update to prevent disadvantageous interactions during other operations (for example package installations)

Certificate management
- The ACME Standalone plugin has improved support for dual-stacked (IPv4 and IPv6) environments and no longer relies on the configured addresses to determine its listening interface

Breaking Changes

Pool permissions

Added new permission Pool.Audit to allow users to see pools, without permitting them to change the pool. The old permission Pool.Allocate now only allows users to edit pools, not to see them. Therefore, Pool.Audit must be added to custom roles with the old Pool.Allocate to preserve the same behavior. All built-in roles are updated automatically.

VZDump
- Hookscript: The TARFILE environment variable was deprecated in Proxmox VE 6, in favor of TARGET. In Proxmox VE 7, it has been removed entirely and thus, it is not exported to the hookscript anymore.
- The size parameter of vzdump has been deprecated, and setting it is now an error.

API deprecations, moves and removals
- The upgrade parameter of the /nodes/{node}/(spiceshell|vncshell|termproxy) API method has been replaced by providing upgrade as cmd parameter.
- The /nodes/{node}/cpu API method has been moved to /nodes/{node}/capabilities/qemu/cpu
- The /nodes/{node}/ceph/disks API method has been replaced by /nodes/{node}/disks/list
- The /nodes/{node}/ceph/flags API method has been moved to /cluster/ceph/flags
- The db_size and wal_size parameters of the /nodes/{node}/ceph/osd API method have been renamed to db_dev_size and wal_dev_size respectively.
- The /nodes/<node>/scan/usb API method has been moved to /nodes/<node>/hardware/usb

CIFS credentials have been stored in the namespaced /etc/pve/priv/storage/<storage>.pw instead of /etc/pve/<storage>.cred since Proxmox VE 6.2 - existing credentials will get moved during the upgrade allowing you to drop fallback code.

qm|pct status <VMID> --verbose, and the respective status API call, only include the template line if the guest is a template, instead of outputting template: for guests which are not templates.

Note that there is no API-stability guarantee during the beta.

Known Issues

Network: Due to the updated systemd version, and for most upgrades, the newer kernel version (5.4 to 5.11), some network interfaces might change upon reboot:
- Some may change their name. For example, due to newly supported functions, a change from enp33s0f0 to enp33s0f0np0 could occur.
  We observed such changes with high-speed Mellanox models.
- Bridge MAC address selection has changed in Debian Bullseye - it is now generated based on the interface name and the machine-id (5) of the system.
Systems installed using the Proxmox VE 4.0 to 5.4 ISO may have a non-unique machine-id. These systems will have their machine-id re-generated automatically on upgrade, to avoid a potentially duplicated bridge MAC.

If you do the upgrade remotely, make sure you have a backup method of connecting to the host (for example, IPMI/iKVM, tiny-pilot, another network accessible by a cluster node, or physical access), in case the network used for SSH access becomes unreachable, due to the network failing to come up after a reboot.

Misc. bug fixes and improvements

Notes

↑ ^1.0 ^1.1 https://github.com/LINBIT/linstor-proxmox

Proxmox VE 6.4

Released 28. April 2021

Based on Debian Buster (10.9)
Ceph Octopus 15.2.11 and Ceph Nautilus 14.2.20
Kernel 5.4 default
Kernel 5.11 opt-in
LXC 4.0
QEMU 5.2
ZFS 2.0.4 - new major version

Virtual Machines (KVM/QEMU):
- Support pinning a VM to a specific QEMU machine version.
- Automatically pin VMs with Windows as OS type to the current QEMU machine on VM creation.
This improves stability and guarantees that the hardware layout can stay the same even with newer QEMU versions.
- Address issues with hanging QMP commands, causing VMs to freeze on disk resize and indeterministic edge cases.
Note that some QMP timeout log messages are still being investigated, they are harmless and only of informative nature.
- cloud-init: re-add Stateless Address Autoconfiguration (SLAAC) option to IPv6 configuration.
- Improve output in task log for mirroring drives and VM live-migration.

Container
- Improved cgroup v2 (control group) handling.
- Support and provide appliance templates for Alpine Linux 3.13, Devuan 3, Fedora 34, Ubuntu 21.04.

Backup and Restore
- Implement unified single-file restore for virtual machine (VM) and container (CT) backup archives located on a Proxmox Backup Server.
The file-restore is available in the GUI and in a new command line tool proxmox-file-restore.
- Live-Restore of VM backup archives located on a Proxmox Backup Server.
No more watching the task log, waiting for a restore to finish; VMs can now be brought up while the restore runs in the background.
- Consistent handling of excludes for container backups across the different backup modes and storage types.
- Container restores now default to the privilege setting from the backup archive.

Ceph Server
- Improve integration for placement group (PG) auto-scaler status and configuration.
  Allow configuration of the CRUSH-rule, Target Size and Target Ratio settings, and automatically calculating the optimal numbers of PGs based on this.

Storage
- Support editing of backup notes on any CephFS, CIFS or NFS storage.
- Support configuring a namespace for accessing a Ceph pool.
- Improve handling ZFS pool by doing separate checks for imported and mounted.
This separation helps in situations where a pool was imported but not mounted, and executing another import command failed.

Disk Management
- Return partitions and display them in tree format.
- Improve detection of disk and partition usage.

Enhancements in the web interface (GUI)
- Show current usage of host memory and CPU resources by each guest in a node's search-view.
- Use binary (1 KiB equals 1024 B instead of 1 KB equals 1000 B) as base in the node and guest memory usage graphs, providing consistency with the units used in the current usage gauge.
- Make columns in the firewall rule view more responsive and flexible by default.
- Improve Ceph pool view, show auto-scaler related columns.
- Support editing existing Ceph pools, adapting the CRUSH-rule, Target Size and Target Ratio, among other things.

External metric servers:
- Support the InfluxDB 1.8 and 2.0 HTTP(s) API.
- Allow use of InfluxDB instances placed behind a reverse-proxy.

Proxmox VE API Proxy Daemon (pveproxy)
- Make listening IP configurable (in /etc/default/pveproxy). This can help to limit exposure to the outside (e.g. by only binding to an internal IP).
- pveproxy now listens for both IPv4 and IPv6, by default

Installation ISO:
- Installation on ZFS:
  - if booted with legacy BIOS (non-UEFI), now also copy the kernel images to the second VFAT partition (ESP), allowing the system to boot from there with grub, making it possible to enable all ZFS features on such systems.
  - set up the boot-partition and boot-loader to all selected disks, instead of only to the first mirror vdev, improving the experience with hardware where the boot-device is not easily selectable.
- The installer environment attempts to do an NTP time synchronization before actually starting the installation, avoiding telemetry and cluster issues, if the RTC had a huge time-drift.

pve-zsync
- Improved snapshot handling allowing for multiple sync intervals for a source and destination pair.
- Better detection of aborted syncs, which previously caused pve-zsync to stop the replication.

Firewall
- Fixes in the API schema to prevent storing rules with a big IP-address list, which get rejected by iptables-restore due to its size limitations. We recommended you to create and use IP-Sets for that use case.
- Improvements to the command-line parameter handling.

Known Issues

Please avoid using zpool upgrade on the "rpool" (root pool) itself, when upgrading to ZFS 2.0 on a system booted by GRUB in legacy mode, as that will break pool import by GRUB.
See the documentation for determining the bootloader used, if you're unsure.

Setups installed with the Proxmox VE 6.4 ISO are not affected, as there the installer always sets up an easier to handle, vfat formatted, ESP to boot.

See the ZFS: Switch Legacy-Boot to Proxmox Boot Tool article about how to switch over to a safer boot variant for legacy GRUB-booted setups with ZFS as root filesystem.

New default bind address for pveproxy and spiceproxy, unifying the default behavior with Proxmox Backup Server
- With making the LISTEN_IP configurable, the daemon now binds to both wildcard addresses (IPv4 0.0.0.0:8006 and IPv6 [::]:8006) by default.
Should you wish to prevent it from listening on IPv6, simply configure the IPv4 wildcard as LISTEN_IP in /etc/default/pveproxy:

LISTEN_IP="0.0.0.0"
- Additionally, the logged IP address format changed for IPv4 in pveproxy's access log (/var/log/pveproxy/access.log). They are now logged as IPv4-mapped IPv6 addresses instead of:
192.168.16.68 - root@pam [10/04/2021:12:35:11 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 854

the line now looks like:

::ffff:192.168.16.68 - root@pam [10/04/2021:12:35:11 +0200] "GET /api2/json/cluster/tasks HTTP/1.1" 200 854

If you want to restore the old logging format, also set LISTEN_IP="0.0.0.0"

Resolving the Ceph `insecure global_id reclaim` Health Warning
With Ceph Octopus 15.2.11 and Ceph Nautilus 14.2.20 we released an update to fix a security issue (CVE-2021-20288) where Ceph was not ensuring that reconnecting/renewing clients were presenting an existing ticket when reclaiming their global_id value.

Updating from an earlier version will result in the above health warning.

See the forum post here for more details and instructions to address this warning.

Proxmox VE 6.3

Released 26. November 2020

Based on Debian Buster (10.6)
Ceph Octopus 15.2.6 (first stable release) and Ceph Nautilus 14.2.15
Kernel 5.4 LTS
LXC 4.0
QEMU 5.1
ZFS 0.8.5

Proxmox Backup Server Integration
- Stable Proxmox Backup Server integration: The stable version 1.0 of Proxmox Backup Server is now integrated and enterprise support is available from the Proxmox support team.
- Data encrypted on client-side before backing up to Proxmox Backup Server.

Ceph
- Stable integration of Ceph Octopus.
- Add selector to choose which supported Ceph version to install in the GUI configuration wizard.
- Recovery progress is displayed in the Ceph status panel.
- Show and allow setting of Placement Group (PG) auto-scaling mode of Ceph pools.
- Set device class when creating OSDs, especially if the auto-detection yields the wrong class.

Enhancements in the GUI
- Improved VM boot device order editor:
  - It is now possible to select multiple devices per type (disk, network) for booting.
  - Booting from passed through PCI devices (e.g., NVMe drives) is supported.
  - Improved user experience with a drag-and-drop UI.
- GUI for editing external metric servers: You can now connect your Proxmox VE nodes to InfluxDB or Graphite using the GUI, instead of having to manually edit /etc/pve/status.cfg
- Optional TLS certificate verification for LDAP and AD authentication realms.
- Improve high-DPI display and browser zoom compatibility.
- Split up storage content view by type.
- Backup/Restore:
  - Overview of all guests, which aren't included in any backup at all.
  - Detailed view per backup job, showing all covered guests and which of their disks are backed up.
- Display optional comments for all storage types.
  - Proxmox Backup Server additionally displays the verification state of all backup snapshots.
- Better usability for preventing accidental snapshot rollback
  - The GUI now makes it difficult to accidentally confuse snapshot removal with snapshot rollback.

Storage
- Add highly flexible backup retention with "keep" settings: The new backup retention settings, which augment and replace the "Max Backups" setting, enable you to decide how many backups to keep per timeframe and implement enhanced retention policies per storage or backup job.
- Better handling of container volume activation on ZFS.
- Increased timeout for connecting to CIFS and NFS storage over slow links.
- Improve querying SSD wear leveling.
- Small improvements to the configuration handling of the LIO iSCSI provider for ZFS over iSCSI storage.
- ZFS disk management: create a systemd service to unconditionally import a pool created using the GUI.

Container
- Add support for current Devuan containers.
- Add support for Kali Linux containers.
- Update list of supported releases for Ubuntu, Fedora and CentOS.
- Support setting a custom timezone per container.
- Improve startup monitoring.
- Add a debug parameter to pct start, to output the debug log of the container start.
- Support systems with up to 8192 cores.
- Optionally ignore mount points while running pct fstrim.
- Fix aborting of backups on Ceph backed containers with a large IO load, by issuing fsfreeze before snapshotting.

QEMU
- Fast, incremental backups to Proxmox Backup Server using dirty-bitmaps.
- Handle guest shutdowns (power down from within a VM) during backups.
- Improved boot order selection allowing booting from multiple virtual disks and from passed through PCI devices (e.g., NVMe drives).
- Allow pass through of certain older Intel iGPU models with QEMU's 'legacy-igd' mode.
- Support more CPU options, for example SSE4.2
- Better support for hugepages across multiple NUMA nodes.

General improvements for virtual guests
- Improved handling of replicated guests when migrating.

Clustering
- Harden locking in the clustered configuration filesystem (pmxcfs), avoids a possible freeze when joining, messaging or leaving a closed-process-group.

User and permission management
- Improved support for using client certificates/keys when connecting to AD/LDAP realms.
- Optional support for case-insensitive logins with AD/LDAP realms.
- Fine-grained permissions for SDN and CloudInit.
- Better handling of clock jumps for rotating keys.

Firewall
- Improved API for matching ICMP-types.

Documentation
- Clarify qdevice requirements.
- Add section about ZFS pool design choices.
- Add documentation on requirement for encrypted ZFS datasets as storage.
- Add manpage for cpu-models.conf(5).

Installer
- Reboot automatically upon successful installation.
- Drop ext3 as supported file system.
- Start a shell on vt3 for debugging during installation.

Experimental: support for Software-Defined Networking (SDN)
- Support for IPAM with a plugin framework.
- Add support for internal IPAM management and PowerDNS.

Countless bug fixes and smaller improvements

Known Issues

VM guests with multiple boot disk setups (e.g., mdadm, LVM, etc...) need to be configured with the new boot order config so that all required disks are marked "bootable", with the one containing the boot loader placed first - otherwise, the guest may fail to boot after being restarted with the new QEMU 5.1 version.
The "exclude-path" option for vzdump now supports non-anchored paths for all backup modes. Non-anchored paths do not start with a '/' and will match in any subdirectory. Previously, such paths only had an effect for "suspend" mode backups, so please ensure that you don't have any unwanted non-anchored paths configured.

Proxmox VE 6.2

Released 12. May 2020

Based on Debian Buster (10.4)
Ceph Nautilus (14.2.9)
Kernel 5.4 LTS
LXC 4.0
Qemu 5.0
ZFS 0.8.3

Web interface
- Allow to deploy DNS based Let's Encrypt (ACME) certificates
- Allow admins to see the full granted permission/privilege tree for a user
- Add experimental Software Defined Network (SDN) GUI and documentation, based on modern mainline Linux Kernel network technology
- Allow to collapse the Notes sub-panel in the guest summary panels. One can do this permanently or automatically if empty through the user settings.
- Add 'Creation Date' column for storage content, allowing one, for example, to comfortably find backups from a certain date
- Add device node creation to Container feature selection
- Full support for up to 8 corosync links
- Automatic guest console reload on VM/CT startup
- Add Arabic translation
- Allow seamlessly changing the language, without logging out and logging in again

Container
- LXC 4.0.2 and lxcfs 4.0.3 with initial full cgroupv2 support by Proxmox VE
- Improve support for modern systemd based Containers
- Improve default settings to support hundreds to thousands* of parallel running Containers per node (* thousands only with simple distributions like Alpine Linux)
- Allow creating templates on directory-based storage

Backup/Restore
- Support for the highly efficient and fast Zstandard (zstd) compression

Improvements to the HA stack
- Allow to destroy virtual guests under HA control when purge is set

QEMU
- Fixed EFI disk behavior on block-based storage (see known issues)
- VirtIO Random Number Generator (RNG) support for VMs
- Custom CPU types with user-defined flags and QEMU/KVM settings
- Improved machine type versioning and compatibility checks
- Various stability fixes, especially for backups and IO-Threads
- Migration:
  - Enable support for Live-Migration with replicated disks
  - Allow specifying target storage for offline mode
  - Allow specifying multiple source-target storage pairs (for now, CLI only)
  - Improve behavior with unused disks
  - Secure live-migration with local disks

General improvements for virtual guests
- Handle ZFS volumes with non-standard mount point correctly

Cluster
- Improve lock contention during high frequency config modifications
- Add versioning for cluster join
- Enable full support for up to 8 corosync links

Ceph
- Easier uninstall process, better informing which actions need to be taken.

Storage
- Storage migration: introduce allow-rename option and return new volume ID if a volume with the same ID is already allocated on the target storage
- Support the 'snippet' content type for GlusterFS storage
- Bandwidth limitations are now also available for SAMBA/CIFS based storage
- Handle ZFS volumes with non-standard mount point correctly
- Improve metadata calculation when creating a new LVM-Thin pool
- Improve parsing of NVMe wearout metrics

User and permission management
- LDAP Sync users and groups automatically into Proxmox VE
- LDAP mode LDAP+STARTTLS
- Allow to add and manage authentication realms through the 'pveum' CLI tool
- Full support and integration for API Tokens
  - Shared or separated privileges
  - Token lifetime
  - Revoke it anytime without impacting your user login

Documentation
- Update hypervisor migration guide in the wiki
- Reducing number of Ceph placement groups is possible
- Improve layout, use a left-side based table-of-content

Various improvements
- Firewall: make config parsing more robust and improve ICMP-type filtering

Known Issues with OVMF/UEFI disks of VMs

A VM's EFI disk on a storage which doesn't allow for small (128 KB) images (for example: CEPH, ZFS, LVM(thin)), was not correctly mapped to the VM. While fixed now, such existing setup may need manual intervention:

You do not have to do anything if your EFI disks is using qcow2 or "raw" on a file based storage.
Before the upgrade, make sure that on your ESP, the EFI boot binary exists at \EFI\BOOT\BOOTX64.EFI (the default EFI Boot fallback).
- Windows and some Linux VMs using systemd-boot should do that automatically
If you already upgraded and it does not boot, see OVMF/UEFI Boot Entries on how to recreate the boot entry via the OVMF Boot Menu

Proxmox VE 6.1

Released 4. December 2019: See Downloads, updated ISO release 2 with installer bug fixes and updated packages on 6. March 2020

Based on Debian Buster (10.2)
Ceph Nautilus (14.2.4.1)
Corosync 3.0
Kernel 5.3
LXC 3.2
Qemu 4.1.1
ZFS 0.8.2

Web interface
- More datacenter configuration options are now editable via GUI:
  - U2F two-factor settings.
  - Cluster-wide bandwidth limit for the following traffic types: migration, backup-restore, clone, disk-move.
- Improvements to two-factor authentication allowing to use a hardware-based TOTP key.
- Mobile GUI: implemented login for TOTP-based two-factor authentication enabled user accounts.
- More icons have been converted from bitmap to vectorized icons from Font Awesome.
- The scale mode of noVNC can be changed in "My Settings".
- Addition of Hebrew language translations.
- New "Run now" button for cluster-wide backup jobs.
- With ifupdown2 installed, it is now possible to change the network configuration and reload it from the GUI without a reboot.

Container
- Pending changes for containers. It is possible to make changes to a running container. They will be applied after the next reboot of the container.
- Reboot of a running container via GUI, API, and command-line interface.
- Hot plugging of mount points by using the new mount API available with Linux Kernel 5.3.
- Support of recent releases of GNU/Linux distributions like Fedora 31, CentOS 8, and Ubuntu 19.10.

SPICE
- Audio devices can now be added via GUI (no need to edit the configuration file).
- Folders can now be shared between SPICE client and VM (still experimental).
- Video streaming can be enabled. This can help to improve the performance of fast-changing display areas like videos.
- The SPICE USB device now supports USB3 (Qemu >= 4.1).

Backup/Restore
- VMs with IOThreads enabled can now be backed up.
- Run scheduled backup jobs manually from the Datacenter in the GUI.

Improvements to the HA stack
- New 'migrate' shutdown policy. If enabled, running services will be migrated to another node on shutdown. Once the node comes back, and if the services have not been manually moved to another node in the meantime, those services will move back.
- New 'crm-command stop'. Shuts down a VM/container with the specified timeout and performs a hard stop if the timeout is 0. A VM or container stop command will now be redirected to this new crm-command.

QEMU
- Allow domains other than '0000' for PCI(e) passthrough.
- New 'reboot' API call. This allows to apply pending changes without having to wait for the guest to shutdown before starting it again.
- QEMU monitor timeout issue, which prevented successful backups in certain configurations, has been fixed.
- PCI(e) passthrough supports up to 16 PCI(e) devices
- Support for QEMU Guest Agents using an ISA serial port (not VirtIO) for communication, allows to use the QGA with FreeBSD - among others.

General improvements for virtual guests
- Allow 'tags' in the guest configuration. This meta-information can be useful for things like configuration management (no GUI support yet).
- VM/CT: Purge allows to remove the corresponding virtual machine or container from replication or backup jobs on destruction.

Cluster stability
- A number of bugs have been identified and fixed upstream in cooperation with corosync and kronosnet.
- The initial issues some users have experienced with MTU changes are resolved.
- pmxcfs has undergone a stability audit using ASAN (AddressSanitizer) and UBSAN (Undefined Behavior Sanitizer), fixing various possible issues for certain edge-cases.

Storage
- Allow configuration of non-default 'mountpoint' properties for ZFS.
- Allow .img files as alternative to .iso images.
- Various iSCSI improvements.
- Overhaul of ZFS on iSCSI with the LIO target provider.
- Supports all features offered by newer kernels with Ceph and KRBD.

Various improvements
- Firewall: Added support for 'raw' tables and use them to allow users to enable and configure Synflood-Attack protections.
- Automatic certificate renewal for the self-signed certificate 2 weeks before expiration.
- Reduced certificate lifetime of newly generated certificates (2 years instead of 10 years). Avoids that some modern Browser complain about very long certificate lifetime.

Documentation
- Proofreading of parts of the documentation (style and grammar).
- Enhanced documentation for the Ceph administration.

Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 6.0

Released 16. July 2019: See Downloads

Based on Debian Buster 10.0
Pre-upgrade checklist tool `pve5to6` - available on Proxmox VE 5.4 and V 6.0
- Running `pve5to6` checks for common pitfalls known to interfere with a clean upgrade process.
Corosync 3.0.2 using Kronosnet as transport
- Default transport method now uses unicast, this can simplify setups where the network had issues with multicast.
- New Web GUI Network selection widget avoids making typos when choosing the correct link address.
- Currently, there is no multicast support available (it's on the kronosnet roadmap).
LXC 3.1
Ceph Nautilus 14.2.x
- Better performance monitoring for rbd images through `rbd perf image iotop` and `rbd perf image iostat`.
- OSD creation, based on ceph-volume: integrated support for full disk encryption of OSDs.
- More robust handling of OSDs (no more mounting and unmounting to identify the OSD).
- ceph-disk has been removed: After upgrading it is not possible to create new OSDs without upgrading to Ceph Nautilus.
- Support for PG split and join: The number of placement groups per pool can now be increased and decreased. There is even an optional plugin in ceph-manager to automatically scale the number of PGs.
- New messenger v2 protocol brings support for encryption on the wire (currently this is still experimental).
- See https://docs.ceph.com/en/nautilus/releases/nautilus/ for the complete release notes.
Improved Ceph administration via GUI
- A cluster-wide overview for Ceph is now displayed in the 'Datacenter View' too.
- The activity and state of the placement groups (PGs) is visualized.
- The version of all Ceph services is now displayed, making detection of outdated services easier.
- Configuration settings from the config file and database are displayed.
- You can now select the public and cluster networks in the GUI with a new network selector.
- Easy encryption for OSDs with a checkbox.
ZFS 0.8.1
- Native encryption for datasets with comfortable key-handling by integrating the encryption directly into the `zfs` utilities. Encryption is as flexible as volume creation and adding redundancy - the gained comfort w.r.t. dm-crypt is comparable to the difference between mdadm+lvm to zfs.
- Allocation-classes for vdevs: you can add a dedicated fast device to a pool which is used for storing often accessed data (metadata, small files).
- TRIM-support - use `zpool trim` to notify devices about unused sectors.
- Checkpoints on pool level.
- See https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0 and https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.1 for the complete release notes.
Support for ZFS on UEFI and on NVMe devices in the installer
- You can now install Proxmox VE with its root on ZFS on UEFI booted systems.
- You can also install ZFS on NVMe devices directly from the installer.
- By using `systemd-boot` as bootloader all pool-level features can be enabled on the root pool.
Qemu 4.0.0
- Live migration of guests with disks backed by local storage via GUI.
- Added support for more Hyper-V enlightenments improving Windows performance in a virtual machine under Qemu/KVM.
- Mitigations for the performance impact of recent Intel CPU vulnerabilities.
- More VM CPU-flags can be set in the web interface.
- Newer virtual PCIe port hardware for machine type q35 in version >= 4.0. This fixes some passthrough issues.
- Support for custom Cloudinit configurations:
  - You can create a custom Cloudinit configuration and store it as snippet on a storage.
  - The `qm cloudinit dump` command can be used to get the current Cloudinit configuration as a starting point for extensions.
Firewall improvements
- Improved detection of the local network so that all used corosync cluster networks get automatically whitelisted.
- Improved firewall behavior during cluster filesystem restart, e.g. on package upgrade.
Mount options for container images
- You can now set certain performance and security related mount options for each container mountpoint.
Linux Kernel
- Updated 5.0 Kernel based off the Ubuntu 19.04 "Disco" kernel with ZFS.
- Intel in-tree NIC drivers are used:
  - Many recent improvements to the kernel networking subsystem introduced incompatibilities with the out of tree drivers provided by Intel, which sometimes lag behind on support for new kernel versions. This can lead to a change of the predictable network interface names for Intel NICs.
Automatic cleanup of old kernel images
- Old kernel images are not longer marked as NeverAutoRemove - preventing problems when /boot is mounted on a small partition.
- By default the following images are kept installed (all others can be automatically removed with `apt autoremove`):
  - the currently running kernel
  - the version being newly installed on package updates
  - the two latest kernels
  - the latest version of each kernel series (e.g. 4.15, 5.0)
Guest status display in the tree view: Additional states for guests (migration, backup, snapshot, locked) are shown directly in the tree overview.
Improved ISO detection in the installer: The way how the installer detects the ISO was reworked to include more devices, alleviating problems of detection on certain hardware.
Pool level backup: It is now possible to create a backup task for backing up a whole pool. By selecting a pool as backup target instead of an explicit list of guests, new members of the pool are automatically included, and removed guests are automatically excluded from the backup task.
New User Settings and Logout menu.
Automatic rotation of authentication key every 24h: by limiting the key lifetime to 24h the impact of key leakage or a malicious administrator are reduced.
The nodes Syslog view in the GUI was overhauled and is now faster.
Sheepdog is no longer maintained, and thus not supported anymore as Storage plugin.
`ceph-disk` has been removed in Ceph Nautilus - use `ceph-volume` instead.
Improved reference documentation
Upgrade documentation: https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0
Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 6.0 beta1

Released 04.07.2019: See Downloads

Based on Debian Buster 10.0
Kernel 5.0.15
ZFS on EFI and/or NVMe installation
Ceph v14.2.1 Nautilus
ZFS 0.8.1
QEMU 4.0.0
LXC 3.1.0
Corosync 3
Misc. bug fixes and improvements

Proxmox VE 5.4

Released 11.04.2019: See Downloads

Based on Debian Stretch 9.8
Kernel 4.15.18
QEMU 2.12.1 (QEMU 3.0.1 since May 2019 via apt)
LXC 3.1.0
ZFS 0.7.13
Ceph 12.2.11 (Luminous LTS, stable), packaged by Proxmox
- Ceph can now be installed via the GUI, thus closing the gap of needing the CLI for this initial step, before configuring a complete (hyperconverged) Ceph setup via PVE's GUI.
QDevice support via `pvecm`
- primarily for small 2-node clusters adding a qdevice can help mitigate the downside of not being able to reboot one node without losing quorum (and thus the ability to make any changes in the cluster) ** Can also help in clusters with a larger even number of nodes by providing a tie-break vote.
- Integration into pvecm and PVE stack vastly simplifies adding a qdevice (it was possible manually before as well)
Wake On Lan support via `pvenode`
- It is now possible to associate a MAC-Address with a PVE-node.
- This can be used to send a WOL (wake on lan) packet to that node from any other node in the PVE-Cluster, and thus power it on on-demand
- Provides a comfortable way of saving power by keeping only part of the nodes permanently online, and starting others on demand (e.g. increased load, or for running guests during upgrades to the PVE-nodes.)
GUI Container wizard creates unprivileged containers by default
HA improvements and added flexibility
- It is now possible to set a datacenter wide HA policy which can change the way guests are treated upon a Node shutdown or reboot. The choices are:
  - freeze: always freeze servivces, independent of the shutdown type (reboot, poweroff)
  - failover: never freeze services, this means that a service will get recovered to another node if possible and if the current node does not comes back up in the grace period of 1 minute.
  - default: this is the current behavior, freeze on reboot but do not freeze on poweroff
Suspend to disk/hibernate support for Qemu/KVM guests
- qemu guests can be 'hibernated' (have their RAM contents and internal state saved to permanent storage) and resumed on the next start.
- This enables users to preserve the running state of the qemu-guests across most upgrades to and reboots of the PVE-node.
- Additionally it can speed up the startup of guests running complex workloads/ workloads which take lots of resources to setup initially, but which need not run permanently.
Support for guest (both Qemu and LXC) hookscripts:
- Hook-scripts are small executables which can be configured for each guest, and are called at certain steps of the guest's lifetime ('pre-start', 'post-start', 'pre-stop', 'post-stop').
- This gives Administrators great flexibility in the way they can prepare the environment for the guest (e.g. adding necessary network resources (routes, vlans), firewall-rules, unlocking encrypted files/devices,...) and cleaning them up when the guest is shutdown or stopped.
Improved Qemu Guest creation wizard:
- Certain often requested options (e.g. Machine-type (q35, pc-i440fx), Firmware (Seabios, UEFI), SCSI controller) can now be selected directly in the VM Creation Wizard, and dependent options get set to sensible values directly.
Clusterwide Subscription Check and Display
- The Datacenter Summary page now gives Administrators a global overview of the subscription status of all nodes in a PVE cluster, thus helping to verify the support-status of the complete cluster on one quick glance
Improved UI in the installer
- It is now possible to go back to a previous screen in the pve-installer, and adapt choices made there, without the need to restart the complete installation process
- Before the actual installation is performed an information-page is displayed containing all relevant information
U2F Authentication for the WebUI
- PVE's WebUI now can optionally use two-factor authentication for users.
- This is generally considered good security practice, and in certain domains and environments it is required.
- The new U2F authentication and the TOTP second factor authentication can be configured by each user for themselves (before all 2FA solutions needed to be configured by a user with User.Modify permission).
Improved reference documentation
Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 5.3

Released 4.12.2018: See Downloads, updated ISO image with installer and apt bug fixes and updated packages on 23.1.2019

Based on Debian Stretch 9.6
Kernel 4.15.18
QEMU 2.12.1
LXC 3.0.2
ZFS 0.7.12
Ceph 12.2.8 (Luminous LTS, stable), packaged by Proxmox
Installer with ZFS: no swap space is created by default, instead an optional limit of the used space in the advanced options can be defined, thus leaving unpartitioned space at the end for a swap partition.
Disk Management on GUI (ZFS, LVM, LVMthin, xfs, ext4)
Create CephFS via GUI (MDS)
CephFS Storage plugin (supports backups, templates and ISO images)
LIO support for ZFS over iSCSI storage plugin
ifupdown2 package and partial GUI support
Delete unused disk images on the storage view
Enable/disable the local cursor in noVNC
Enable/disable autoresize in noVNC
Edit /etc/hosts/ via GUI, which is necessary whenever the IP or hostname is changed
Editable VGA Memory for VM, enables higher resolutions (UHD) for e.g. SPICE
VM Generation ID: Users can set a 'vmgenid' device, which enables safe snapshots, backup and clone for various software inside the VM, like Microsoft Active Directory. This vmgenid will be added to all newly created VMs automatically. Users have to manually add it to older VMs.
qemu-server: add ability to fstrim a guest filesystem after moving a disk (during move_disk or migrate with local disks)
Emulating ARM virtual machines (experimental, mostly useful for development purposes)
vGPU/MDev and PCI passthrough. GUI for configuring PCI passthrough and also enables the use of vGPUs (aka mediated devices) like Intel KVMGT (aka GVT-g) or Nvidias vGPUS.
pvesh rewrite to use the new CLI Formatter framework - providing a configurable output format for all API calls (formats include: json, yaml and a pleasant table output with borders). Provides unified support for json output for the PVE CLI Tools.
Nesting for Containers (privileged & unprivileged containers): Allows running lxc, lxd or docker inside containers, also supports using AppArmor inside containers
Mounting CIFS/NFS inside containers (privileged): Allows using samba or nfs shares directly from within containers
Improved reference documentation
Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 5.2

Released 16.05.2018: See Downloads

Based on Debian Stretch 9.4
Kernel 4.15.17
QEMU 2.11.1
LXC 3.0.0
Ceph 12.2.5 (Luminous LTS, stable), packaged by Proxmox
ZFS 0.7.8
Cloudinit GUI support
Cluster create/join nodes via GUI
Certificate management including Let´s Encrypt GUI
SMB/CIFS Storage plugin (supports backups, images, templates, iso and containers)
Display IP for VM (using qemu-guest-agent)
LXC: templates and clones, move volume/disk
Create and edit new roles via GUI
I/O bandwith limits for restore operations (globally, per storage or per restore job)
new and improved xterm.js integration including reconnect logic (on container reboots or restart migrations)
Basic/Advanced GUI
ebtables support
Improved reference documentation
Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 5.1

Released 24.10.2017: See Downloads, updated ISO image with installer bug fixes and updated packages on 18.12.2017

Based on Debian Stretch 9.2
Kernel 4.13.3
QEMU 2.9.1
LXC: update to 2.1
Ceph 12.2.1 (Luminous LTS, stable), packaged by Proxmox
ZFS 0.7.2
Improved reference documentation with screenshots
Countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 5.0

Released 04.07.2017: See Downloads, updated ISO image with installer bug fixes and updated packages on 09.08.2017

Based on Debian Stretch 9.0
Kernel 4.10.15
QEMU 2.9
LXC: update to 2.0.8
New asynchronous Storage Replication feature (needs ZFS, technology preview)
New/updated LXC templates (Debian, Ubuntu, CentOS, Fedora, OpenSUSE, Arch Linux, Gentoo and Alpine)
Updated/improved noVNC console]
Ceph v12.1.0 Luminous (technology preview), packaged by Proxmox
live migration with local storage
GUI improvements
- USB und Host PCI address visibility
- improved bulk and filtering options
Improved installation ISO
Importing Qemu/KVM_Virtual_Machines#_importing_virtual_machines_from_foreign_hypervisors
improved reference documentation with screenshots
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 5.0 beta2

Released 23.05.2017: See Downloads

Based on Debian Stretch
Kernel 4.10
Update to QEMU 2.9
Improved handling of re-used disks in installer
Improved handling of network devices in installer
Many small improvements and bug fixes

Proxmox VE 5.0 beta1

Released 22.03.2017: See Downloads

Based on Debian Stretch
Kernel 4.10
Many small improvements and bug fixes

Proxmox VE 4.4

Released 13.12.2016: See Downloads

update kernel to 4.4.35
KVM: update to qemu 2.7.0
LXC improvements
- update to LXC 2.0.6
- implement new restart migration
- unprivileged containers (now the GUI)
- updated template repository (Debian, Ubuntu, CentOS, Fedora, Arch and Alpine)
GUI improvements
- new ceph dashboard
- new cluster dashboard
- improved disk management, smart status supporting more SSD types
- HA improved GUI
ISO installer supports advanced ZFS settings
dedicated network for VM/CT migrations (CLI only)
improved reference documentation, e.g. contains now also screenshots
DRBD9 (removed from core, now maintained directly by Linbit, due to license change)
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.3

Released 27.09.2016: See Downloads

GUI improvements
- new search (“ctrl-shift-f”)
- vertical menus with groups and icons
- double click to open VM/CT console
- new status overview for host, VM and containers, signal colors for overloaded CPU, etc.
- Task log window remembers its size
VM wizard proposes now optimal settings dependent to the selected operating system (for Linux the default is “virtio scsi disk” and "virtio net")
comprehensive reference documentation in html, pdf and epub format, including contextual help function on several places - based on asciiDoc
new disk management, including SMART monitoring and SSD wearout level display (for Intel, Sandisk and Samsung SSD).
based on Debian Jessie 8.6.0
Linux kernel 4.4.19
KVM/qemu 2.6.1
LXC: update to 2.0.4
many new and improved manual/help pages
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.2

Released 27.04.2016: See Downloads

GUI update to Sencha Ext JS 6, including new icon set and dynamic performance charts
based on Debian Jessie 8.4.0
Linux kernel 4.4.6
KVM/qemu 2.5.1
installer: LVM thin or ZFS plugin pre-configured
LVM thin support
Let´s Encrypt support
DRBD9: drbd 9.0.2 kernel module and drbdmanage 0.95
LXC: improved container setup
LXC: network rate limits
LXC: adding mount-points via GUI
LXC: pct can mount rootfs for read/write access of stopped or running containers
LXC: backup/snapshot improvements, including proper backup and restore of ACL
LXC: add support for Alpine Linux, Ubuntu 16.04
HA manager improvements
new confirmation dialog to prevent accidentally removal of a virtual machine or container
updated GUI translation (especially French, Italian and German)
many new and improved manual pages
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.1

Released 11.12.2015: See Downloads

based on Debian Jessie 8.2.0
Linux kernel 4.2.6
improved startup/shutdown behavior (systemd)
enable NTP by default
installer: allow up to 8 disks for ZFS
KVM: add qemu agent GUI option
KVM: update network boot ROMs
Improve HA GUI for users with restricted permissions
add Galician language to GUI
LXC: add rootfs resize to GUI
LXC: add support for Fedora 22 and Debian stretch/sid, Ubuntu 15.10
LXC: support unpriviledged containers (technology preview)
storage: added LVM thin support (technology preview)
Support for Turnkey Linux LXC appliances
added new pvereport command
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.0

Released 05.10.2015: See Downloads

based on Debian Jessie 8.2.0
Linux kernel 4.2
QEMU 2.4
new high availability manager
bash completion support
full IPv6 support
include LXC (and removed OpenVZ) - see Convert OpenVZ to LXC
added DRBD9 with drbdmanage (technology preview)
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.0 beta2

Released 10.09.2015: See Downloads

based on Debian Jessie 8.2.0
Linux kernel 4.2
QEMU 2.4
LXC improvements Linux Container
- Containers can be created on local storage, NFS, Ceph RBD, DRBD9, ...
- Migration (offline)
IPv6 support
DRBD9
Ceph Server packages
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 4.0 beta1

Released 22.06.2015: See Downloads

based on Debian Jessie 8.1
use kernel 3.19.8
new HA manager, see High_Availability_Cluster_4.x
QEMU 2.3
include LXC (and removed openvz), see Linux Container
DRBD9
countless bug fixes and package updates (for all details see bugtracker and GIT)

Proxmox VE 3.4

Released 19.02.2015: See Downloads - updated ISO image with ZFS 0.6.4.1 on 27.05.2015

add ZFS support
- installer supports all ZFS RAID levels.
pve-manager (GUI) updates
- openvz: implement suspend/resume
- qemu GUI: support new pending changes API
- new 'disconnect' button on the network edit panel
- vzdump: new option --stop to abort running backup job
- add NUMA option to processor form
- add GUI for local ZFS storage
- add startall/stopall/migrateall menu buttons
- add multiqueues field to NIC form

update to qemu 2.1.3

qemu-server updates
- shutdown by Qemu Guest Agent if the agent flag is set
- snapshot_create: call fsfreeze if agent flag is set
- vzdump: use qga freeze in vzdump in snapshot mode
- add custom NUMA topology support
- serial: allow to pass arbitrary device names
- add link_down flag to network config
- support additional e1000 variants for VM machines
- enable hotplug by default
- hotplug config: allow to enable specific features
- include memory hotplug patch (experimental)

improved hotplug support

based on Debian Wheezy 7.8

kernel updates

countless bug fixes and package updates, for all details see bugtracker and GIT

Proxmox VE 3.3

Released 15.09.2014: See Downloads

improved security features
- Firewall support (new package pve-firewall)
- Two-Factor Authentication (Yubico and OATH)

pve-manager (GUI) updates
- new Proxmox VE Firewall
- noVNC console
- openvz: add bridge vlan && firewall options to gui
- new Proxmox VE Mobile, GUI for mobile devices
- add new 'Pool View'
- ZFS storage can now be configured on GUI
- glusterfs: new option to specify backup volfile server
- add new email_from option to datacenter.cfg
- add Persian (Farsi) translation.
- improved Spanish translation
- update Chinese translation
- Countless updates and fixes

update to qemu 2.1.0
- pci passthrough improvements
- hotplug improvements
- migration : enable auto-converge capability
- add cpu_hotplug (and maxcpus config)
- add virtio-net multiqueue support
- new option smbios1 to specify SMBIOS type 1 fields
- set uuid for newly created machines
- support new q35 machine type
- add Broadwell cpu model
- compile with new libiscsi (1.12.0)
- use glusterfs 3.5.2 libraries
- support drive option 'discard'
- add support for new qemu throttling burst max parameters
- add 'vmxnet3', 'lsi53c810', and 'pvscsi' to the list of available network card models

improved Console support
- HTML5 Console for shell, VM and container console (noVNC)
- noVNC console is now the default
- vncterm: new option -notls (for novnc, which use 'wss')
- vncterm: updated signature for java applet to avoid warnings

pve-kernel-2.6.32-32-pve: 2.6.32-136
- update aacraid, arcmsr, netxtreme2, ixgbe, igb, megaraid_sas and e1000e drivers
- update to vzkernel-2.6.32-042stab093.4.src.rpm
- allow to use grub-efi-ia32 boot loader

pve-kernel-3.10.0-4-pve: 3.10.0-17
- enable vfio xfga
- update arcmsr, netxtreme2, ixgbe, igb, e1000e drivers
- update to kernel-3.10.0-123.6.3.el7.src.rpm
- allow to use grub-efi-ia32 boot loader
- Note: there is still no OpenVZ support with this kernel

update ceph packages to 'firefly' (0.80.5)
- Note: Please upgrade ceph packages first if you run ceph server on proxmox nodes (see ceph upgrade instructions).

update gluster packages to 3.5.2

fence-agents-pve: 4.0.10
- update to 4.0.10
- add fence_ovh and fence_amt
- remove baytech, bullpap, cpint, egenera, mcdata, nss_wrapper,rackswitch, vixel, xcat. Those agents are no longer included in upstream package.
- removed fence_scsi
- Note: This includes updates for fence_ipmilan (fence_ilo3, fence_ilo4, fence_imm, and fence_idrac), and some parameter names changed (see 'man fence_ipmilan'). Please verify that your fence device still works if you use HA.

based on Debian Wheezy 7.6
countless bug fixes and package updates, for all details see bugtracker and GIT

Proxmox VE 3.2

Released 10.03.2014: See Downloads

improved SPICE support
- spiceterm: console for OpenVZ and host
- add new console option to datacenter.cfg (java applet vs. spice)
- add multi-monitor support
- GUI: use split-button to easily select SPICE or VNC
- more details on https://pve.proxmox.com/wiki/SPICE
update qemu to 1.7.0
- add 'pvscsi' to the list of scsi controllers (emulate the VMware PVSCSI device)
- add 'lsi53c810' to the list of scsi controllers (supported on some very old Windows NT versions)
- add 'vmxnet3' to the list of available network card models (emulate VMware paravirtualized network card)
- add drive option 'discard'
- add support for new qemu throttling burst max parameters
- improved live backup
pve-kernel-2.6.32-27-pve: 2.6.32-121
- update to vzkernel-2.6.32-042stab084.20.src.rpm
- update e1000, igb, ixgbe, netxtreme2, megaraid_sas
- include latest ARECA RAID drivers
- update Broadcom bnx2/bnx2x drivers to 7.6.62
- update aacraid to aacraid-1.2.1-30300.src.rpm
Ceph Server (Technology Preview)
- new GUI to manage Ceph server running on PVE nodes
- more details on https://pve.proxmox.com/wiki/Ceph_Server
added Open vSwitch support (Technology Preview)
Optional 3.10 Kernel (based on RHEL7 beta, currently without OpenVZ support, for testing only)
storage: new ZFS plugin (Technology Preview), see https://pve.proxmox.com/wiki/Storage:_ZFS
storage: remove nexenta plugin (ZFS plugin is faster)
updated GlusterFS to 3.4.2
ISO installer uses now always GPT partition table
- added 'gdisk' to manage and view partitions via CLI
based on Debian Wheezy 7.4
countless bug fixes and package updates (for all details see bugtracker and GIT

Proxmox VE 3.1

Released 21.08.2013: See Downloads

pve-kernel-2.6.32-23-pve (2.6.32-109)
- update to vzkernel-2.6.32-042stab079.5.src.rpm
- remove fix-ipoib-add-missing-lock.patch (now upstream)
- include Highpoint 2710 RAID driver
- update megaraid_sas to 6.600.18.00
pve-cluster (3.0-7)
- suppress another warning about uninitialized value at boot time
- avoid warning when adding a new node (can't create shared ssh key database...)
- avoid warning if /etc/pve/priv/ already exists
- add migration_unsecure param to datacenter.cfg
pve-firmware (1.0-23)
- update for RHEL6.4 based kernels
pve-manager (3.1-3)
- add new favicon.ico
- fix perl dependencies
- display subscription status after login
- add GUI for GlusterFS storage
- vzdump: wait one second before unmounting the snapshot (avoid device busy error)
- apt: try to detect new packages
- tell apt to use http_proxy settings
- support to view changelogs for available pve packages
- allow to configure cipher list for pveproxy
- automatically add pve repository to apt sources list
- allow to use serial terminal as display
- avoid apt autoremoval for pve-kernel packages
- pveproxy: use same cipher list as debian apache ssl.conf
- avoid apt autoremoval for pve-kernel packages
- avoid and fix several proxy loops
- correctly initialize $rpcenv in spiceproxy
- fix bug #437: include pve-libspice-server1 in package list
- improved spice proxy code
- add button to start SPICE console
- add spiceproxy server
- fix upgrade shell (quoting bug)
- new pveupgrade script, used to display additional information to the user (reboot required? database up to date? ...)
- add button to display package versions, to provide same info as 'pveversion -v' on GUI.
- fix bug #408: reset ENV before starting shell
- fix bug #379: allow to overwrite existing CTs if user has VM.Backup permissions
- Allow to send notification mail about new packages.
- package manager: run system upgrade inside VNC shell, group packages by 'Origin', remove apt/upgrade API completely.
- package manager: improve changelog viewer, bug fixes
- add package update manager to GUI
- update Norwegian translations
- rrd display : avoid 2 refresh
- avoid gzip http response for png & gif & jar
pve-qemu-kvm (1.4-17)
- enable glusterfs backend
- add patch to use pce certs for spice by default
- disable spice-socket.patch
- Allow to query spice ticket
- add usbredir support to spice
- add experimental code to support spice
qemu-server (3.1-1)
- allow pass through of usb parallel devices (--parallel0 /dev/usb/lp0)
- fix bugs in migration code (wrong qw() usage)
- vncproxy: load config from correct node
- allow to use a socket for serial devices
- implement 'qm terminal' to open terminal via serial device
- add ability to run without graphic card ('vga: serial[n]')
- add support for unsecure/fast migration (setting in datacenter.cfg)
- remove spice cert paths (depend on pve-qemu-kvm >= 1.4-16)
- implement spice seamless migration
- support usb redirection devices for spice (usb[n]: spice)
- disable tablet device by default for spice
- spiceproxy API: allow client to choose proxy address
- spiceproxy API: read cert subject name directly using Net::SSLeay
- spice: use TLS (encrypt whole traffic)
- allow to pass SCSI generic devices to guests, for example "scsi0: /dev/sg5"
- cpu flags optimization
- add support for SPICE
vncterm (1.1-4)
- Allow to add intermediate certificates to /etc/pve/local/pve-ssl.pem (users previously used apache option SSLCertificateChainFile for that).
pve-libspice-server1 (0.12.4-1)
- initial release and update to 0.12.4
ceph (0.67.1-1~bpo70+1)
- New upstream release
libpve-access-control (3.0-6)
- use shorter spiceproxy tickets
- add code to generate tickets for SPICE
libpve-common-perl (3.0-6)
- auth.conf: avoid warning if file does not exist, fix file permissions
- add parser for "/etc/apt/auth.conf"
libpve-storage-perl (3.0-10)
- add Glusterfs Plugin
- API: add scan method for glusterfs
- sheepdog: fixes for storeid != 'sheepdog'
lvm2 (2.02.98-pve4)
- New upstream release (fix for udev rules)
glusterfs
- support for glusterfs (technology preview, https://pve.proxmox.com/wiki/Storage:_GlusterFS)
SPICE
- support for Spice protocol (technology preview, https://pve.proxmox.com/wiki/SPICE)
based on Debian 7.1 (Wheezy)
Countless bug fixes (for all details see bugtracker and GIT

Proxmox VE 3.0

Released 24.05.2013: See Downloads

pve-kernel-2.6.32 2.6.32-100
- fix CVE-2013-2094
update ceph packages to 0.61.2
libpve-common-perl 3.0-4:
- fix bug #381: use persistent reservation file for ports
- new function PVE::Tools::next_migrate_port()
libpve-storage-perl 3.0-6
- rbd : clone volume to same pool that base volume
- extend storage list API with useful parameters for clone
- fix iscsi session scan bug
pve-cluster 3.0-4
- depend on fuse (seems most utilities moved from fuse-utils to fuse)
- remove depencency on fuse-utils (this package is no longer required)
- fix warning about uninitialize value
- fix bug 383: restart pveproxy instead of apache2
pve-manager 3.0-20
- fix login problem when http_proxy is set
- updated Chinese, German and French translation
- allow to upload files with spaces in filename (replace with '_')
- allow to delete pools with non-existent VMs/Storage
- use correct icons for templates
- sort templates after regular VMs
- StorageSelector: allow to use new target option, used by Clone
- fix bug #385: correctly handle Accept-Encoding
- fix bug #374: correctly remove destroyed container from pool
pve-qemu-kvm 1.4-12
- vma create: only store basename of config file.
qemu-server 3.0-15
- clone disk : keep source volume params
- clone: check is we can clone to target storage
- fix bug #381: use PVE::Tools::next_migrate_port()
- restore: do not restore template flag
vncterm 1.1-3
- re-enable javascript-events.patch (needed by migrate)
based on Debian 7.0 (Wheezy)
new VM clone feature
new event driven API server (pveproxy)
- completely replace apache2
- efficient support for HTTP keep-alive
support bootlogd (boot log can be viewed on the GUI)
update pve-qemu-kvm to 1.4.1
update kernel to vzkernel-2.6.32-042stab076.7.src.rpm
- changed default IO Scheduler to 'deadline'
- updated Intel network drivers for e1000e, igb and ixgbe
Countless bug fixes (for all details see bugtracker and GIT

Proxmox VE 2.3

Released 04.03.2013: See Downloads

update qemu-kvm to 1.4.0
new kvm backup implementation, see Backup and Restore
added RBD (ceph) support on GUI
update kernel to vzkernel-2.6.32-042stab072.10.src.rpm
include latest Broadcom bnx2/bnx2x drivers
include latest Adaptec aacraid driver 1.2-1[29900]
update e1000e to 2.2.14
update igb to 4.1.2
update ixgbe to 3.12.6
enable CONFIG_RT_GROUP_SCHED (also update corosync if you install this kernel)
extend memory GUI to support ballooning
implement auto-ballooning
add HD resize feature to expand disks
updated network drivers (bnx2/bnx2x/e1000e/igb/ixgbe)
added omping binaries (for testing multicast between nodes)
update to latest Debian version 6.0.7
qcow2 as default storage format, cache=none (previously raw)
KVM64 as default CPU type (previously qemu64)
e1000 as default NIC (previously rtl8139)
task history per VM
Node Summary: added "KSM sharing" and "CPU Socket count"
enable/disable tablet for VM on GUI without stop/start of VM (you can use vmmouse instead, for lower CPU usage, works on modern Linux and on all Windows VMs as long as you install the vmmouse drivers)
bug fixes (for all details see bugtracker and GIT

Proxmox VE 2.2

Released 24.10.2012: See Downloads

update kernel to vzkernel-2.6.32-042stab062.2.src.rpm
update Intel nics drivers (e1000e to 2.1.4, ixgbe to 3.11.33, igb to 4.0.17)
update qemu-kvm to 1.2.0
openvz: update vzctl to 4.0
openvz: use real console instead of 'vzctl enter'
add Live Snapshots support (qcow2)
added Slovenian translation
kvm: new option to select SCSI controller hardware
kvm: support up to 32 network devices
kvm: support up to 16 virtio devices
kvm: add SATA to GUI
updated cluster packages
update to latest Debian version 6.0.6
bug fixes (for all details see bugtracker and GIT

Proxmox VE 2.1

Released 27.04.2012: See Downloads

simplify GUI for users without permissions
implemented startup/shutdown ordering
improved task logs on node startup/shutdown
added Russian, Italian and Turkish translation
updated corosync cluster stack to 1.4.3
update LVM to 2.02.95 (fixes snapshot problems)
bug fixes

Proxmox VE 2.0

Updated ISO 5.4.2012: See Downloads

pve-kernel-2.6.32-11-pve (2.6.32-65)
- rebase on vzkernel-2.6.32-042stab053.5.src.rpm
fix for windows timing issues ("args: -no-hpet -no-kvm-pit-reinjection" are now defaults for newer Windows guests)
subscription key management (commercial subscribers)
vzdump fixes (qmrestore)

Released 30.03.2012: See Downloads

(First public beta released on 30.09.2011, rc1 on 16.02.2012, final release on 30.03.2012)

Proxmox VE 2.0 Documentation

Complete new GUI
- based on Ext JS 4 JavaScript framework
- fast search-driven interface, capable of handling hundreds and probably thousands of VM´s
- secure VNC console, supporting external VNC viewer with SSL support
- role based user- and permission management for all objects (VM´s, storages, nodes, etc.)
- Support for multiple authentication sources (MS ADS, LDAP, Linux PAM, Proxmox VE authentication)
High Availability Cluster support for KVM guests and OpenVZ containers
- resource agents for KVM and OpenVZ
- GUI for managing KVM and OpenVZ HA settings
Based on Debian 6.0 Squeeze
- long-term 2.6.32 Kernel (based on RHEL6.x) with KVM and OpenVZ as default
New cluster communication based on corosync, including:
- Proxmox Cluster file system (pmxcfs): Database-driven file system for storing configuration files, replicated in realtime on all nodes using corosync
- creates multi-master clusters (no single master anymore!)
- cluster-wide logging
- basis for HA setup´s with KVM guests and OpenVZ containers config files
RESTful web API Proxmox_VE_API
- Resource Oriented Architecture (ROA)
- declarative API definition using JSON Schema
- enable easy integration for third party management tools
Backup/Restore via GUI
- GUI and CLI, works for OpenVZ containers and KVM VM´s
- "Backup Now" via GUI
- Restore via GUI
- All jobs can be monitored as “Recent tasks”
OpenVZ
- Appliance Downloader, including TurnKey Linux channel
- Multiple storages for OpenVZ container, no limit to /var/lib/vz anymore!!!
- vswap support
- Improved init.log (shows start-up logs of a OpenVZ container)
Commitment to Free Software (FOSS): public code repository and bug tracker for the 2.x code base.
Licensed under GNU Affero General Public License, version 3 https://www.gnu.org/licenses/agpl-3.0.html.

Proxmox VE 1.9

Updated ISO 27.9.2011, including the following, see Downloads

Release Notes:

pve-kernel-2.6.32-6-pve (2.6.32-47)
- update to vzkernel-2.6.32-042stab037.1.src.rpm
- use bnx from upstream
- update igb driver to 3.2.9
- update e1000e driver to 1.6.2
- enable GDTH driver
- update ata_generic to support intel IDE-R
- use BIOS default settings for ASPM (fix-aspm-policy.patch)
pve-firmware (1.0-14)
- include firmware for new bnx2 drivers
vzctl (3.0.29-2pve1)
- update to 3.0.29.2
vzdump (1.2-16) unstable; urgency=low
- use rsync option '--sparse' on first pass (take less disk space - as suggested by John Stevens)

Original 1.9 ISO, released 13.09.2011: See Downloads

Release Notes:

pve-kernel-2.6.32 (2.6.32-43)
- rebase on vzkernel-2.6.32-042stab036.1.src.rpm (new stable Openvz kernel branch), updates for drivers including e1000e to 1.5.1, ARECA RAID driver, megaraid_sas, bnx2, igb to 3.1.16, ixgbe to 3.5.14-NAPI, drbd 8.3.10, ...
vzctl (3.0.28-1pve5)
- update to latest upstream
- set defaull template to debian-6.0-standard
- set CONFIGFILE="pve.auto" in /etc/vz/vz.conf
- merge some fixes from upstream
pve-manager (1.9-24)
- fix uptime display for 2.6.32 kernel with 1000HZ
- support newer vzctl versions.
- support 'maxfiles' backup option.
pve-qemu-kvm (0.15.0-1)
- update to upstream 0.15.0
- use pxe roms from upstream qemu-kvm
qemu-server (1.1-32)
- small fixes for new qemu-kvm 0.15.0
libpve-storage-perl (1.0-19)
- set LC_ALL instead of LANG (avoid bug when user sets LC_ environment variables)
- iscsi: tolerate errors when not all portals are online.
vzdump (1.2-15)
- run pre-restart hook after snapshot

Proxmox VE 1.8

Updated ISO 16.6.2011, includes current stable packages Downloads

pve-kernel-2.6.32 (2.6.32-33)
aacraid with support for Adaptec 6 series (e.g. 6805)
KVM 0.14.1
vzctl 3.0.27
vzdump 1.2-13
bug fixes

Released 30.03.2011: See Downloads

Release Notes:

update to Debian Lenny 5.0.8
update to kvm 0.14.0
updated kernels with many fixes
protect against Cross Site Request Forgery (added anti-CSRF tokens)
bug fixes

Detailed change log:

qemu-server (1.1-30)
- use 'cache=none' for raw images and devices to improve performance

pve-qemu-kvm (0.14.0-2)
- also install vgabios-qxl.bin vgabios-stdvga.bin vgabios-vmware.bin
- update to 0.14.0
- add fix for eventfd support (msix-eventfd-fix.patch)
- removed kvmtrace (removed from upstream?)
- add vnc keyboard fixes for fr-ca (reported by Pierre-Yves)

pve-kernel-2.6.32 (2.6.32-32)
- update to Debian Squeeze kernel 2.6.32-32
- remove patch use-init-task-credential.patch (fixed in upstream)
- enable CONFIG_SENSORS_K10TEMP=m

pve-kernel-2.6.18 (2.6.18-14)
- update to ovzkernel-2.6.18-238.5.1.el5.028stab085.3.src.rpm
- increased kernel API version to '5-pve'
- update to SCST 2.0.0.1
- update to OCFS2 1.4.7

pve-qemu-kvm-2.6.18 (0.9.1-12)
- update to kvm-83-224.el5.src.rpm

pve-kernel-2.6.35 (2.6.35-10)
- update to Ubuntu-2.6.35-27.48

pve-manager (1.8-15)
- protect against Cross Site Request Forgery (added anti-CSRF tokens)
- correctly encode html entities inside notes (avoid persistant XSS vulnerabilities)
- use http_proxy in cron APL download if configured

Proxmox VE 1.7

Released 30.11.2010: See Downloads

Change log:

libpve-storage-perl (1.0-16)
- several fixes for multipath

pve-kernel-2.6.18 (2.6.18-10)
- update to ovzkernel-2.6.18-194.26.1.el5.028stab070.14.src.rpm

pve-kernel-2.6.32 (2.6.32-28)
- update to Debian Squeeze kernel 2.6.32-28 (openvz 2.6.32-dzhanibekov)
- update igb driver to version 2.4.8
- update ixgbe driver to version 3.0.14

pve-manager (1.7-10)

- we only display the warning about missing openvz if we have no kvm.
- fix config change detection in pvectl
- on the create VM page, query capabilities of remote host (kvm/openvz enabled?) and select resonable defaults.

pve-qemu-kvm (0.13.0-2)
- do not install unnecessary blobs (manually install blobs, use --disable-blobs)
- update migration and vnc keymap patches for 0.13.0

vzdump (1.2-9)
- fix veth numbers when restoring openvz containers (vzrestore)

qemu-server (1.1-25)
- new --repeat option for qmrestore
- use new -netdev syntax for network devices. This enables tcp offload settings, and we can use 'vhost=on' for virtio-net

Small bug fixes

Proxmox VE 1.6 (updated) - ISO Installer with 2.6.32 Kernel with OpenVZ including KVM 0.12.5

Released 18.10.2010: See Downloads

Change log:

pve-kernel-2.6.32 (2.6.32-24)
- include fix for OpenVZ bug #1603
- ixgbe driver update (3.0.12)
- upgrade to Debian Squeeze kernel 2.6.32-24 (OpenVZ 2.6.32-dyomin)
- strip debug info
- remove default initrd image (no longer needed)
- do not set CONFIG_DMAR_DEFAULT_ON
- enable CAPIDRV module

pve-qemu-kvm (0.12.5-2)
- enable up to 32 NICs (up to 29 are usable)

pve-manager (1.6-5)
- updated Japanese translations
- minor fixes in bin/pveversion (adjust for proxmox-ve-2.6.35)
- update Czech translations.

iscsi-scst (1.0.1.1-4)
- create /var/lib/scst/pr (required by the kernel module)

qemu-server (1.1-22)
- use larger block size for dd (bs=256k) to speed up restore.
- new backup=yes|no option for drives. VZDump ignores drives with option backup=no (many thanks to draguz)
- new qmrestore option '--unique': create unique MAC address on restore (many thanks to Seth)

vzdump (1.2-8)
- correctly escape output from 'find' (escape backslash using sed). It is now possible to backup files which contain a backslash character.
- added new hook 'log-end' (see hook-script.pl example)

dab (1.1-9)
- correct update-rc.d invocation (add '.' after runlevel)
- fix postgres startup script for squeeze
- support dependency based startup (squeeze)
- use postgresql 8.4 for squeeze

pve-kernel-2.6.35 (2.6.35-6)
- first public release (KVM only, KSM enabled)

pve-kernel-2.6.18 (2.6.18-8)
- update to ovzkernel-2.6.18-194.17.1.el5.028stab070.7.src.rpm
- remove initrd code (no longer needed)
- strip debug symbols (reduce kernel package size)

pve-qemu-kvm-2.6.18 (0.9.1-8)
- update to kvm-83-164.el5_5.23.src.rpm
- enable all (previously disabled) kvm features (scsi, usb, ...)

pve-firmware (1.0-9)
- include firmware for new 2.6.35 kernels

Small bug fixes

Proxmox VE 1.6 - ISO Installer with 2.6.32 Kernel with OpenVZ including KVM 0.12.5

Released 30.08.2010: See Downloads

Change log:

New 2.6.32 kernel including OpenVZ support, based on latest Debian Squeeze Kernel
Removed KSM (OpenVZ does not support KSM)
DRBD tools: update to drbd 8.3.7
New KVM user-space tools (0.12.5)
New OpenVZ user-space tools (vzctl 3.0.24)
vzdump: use ionice to set io priority. By default, we now set "ionice -c2 -n7" to limit io (instead of using a default bwlimit)
vzrestore: new --force option to overwrite existing data
dab bug fix: only print a warning for packages installed directly with 'dpkg' (when there is no information in 'available' list)
dab: only mount /proc/ if not already mounted (avoid problems with vzctl 3.0.24)
Bug fixes

Update 20.9.2010:

2.6.32 Kernel:

pve-kernel-2.6.32 (2.6.32-19)
include latest Broadcom bnx2, bnx2x drivers
update to Debian kernel 2.6.32-22
fix CVE-2010-3081 kernel
include latest ixgbe driver

2.6.24 Kernel:

pve-kernel-2.6.24 (2.6.24-25)
update to Ubuntu-2.6.24-28.7 (fix CVE-2010-3081)

2.6.18 Kernel:

pve-kernel-2.6.18 (2.6.18-6)
fix CVE-2010-3081

Proxmox VE 1.5 - New Kernel 2.6.24 and 2.6.32, including KVM 0.12.4 and gPXE

Released 17.05.2010: See Downloads

Change log 2.6.24 (KVM and OpenVZ):

update to qemu-kvm-0.12.4
Seabios
include gPXE 1.0 network boot (WDS is working now)
update e1000e to e1000e-1.1.2.tar.gz
update igb to igb-2.1.9.tar.gz
update e1000 to e1000-8.0.19.tar.gz
update bnx2 to netxtreme2-5.2.50.tar.gz
update tg3 to tg3-3.105h.tar.gz
include alt1e (Atheros nic)
kernel update (latest 2.6.24)
Fixed drive order bug (KVM)
Allow up to 4094 bridges
update to the latest vzctl
Use option –sparse for qmigrate

Change log 2.6.32 (KVM only):

update to qemu-kvm-0.12.4
Seabios
include gPXE 1.0 network boot (WDS is working now)
kernel update (latest 2.6.32)
Fixed drive order bug (KVM)
Allow up to 4094 bridges
Use option –sparse for qmigrate

Proxmox VE 1.5

Released 20.01.2010: See Downloads

Including:

many Linux kernel improvements
support different kernel versions
new 2.6.18 Linux kernel - default (OpenVZ, KVM)
new 2.6.24 Linux kernel (OpenVZ, KVM)
new 2.6.32 Linux kernel (KVM only, KSM support)
use update-initramfs
DRBD: update to drbd 8.3.4
AOE: update to AOE6-73
SCST: update to latest SCST modules
new KVM user-space tools
we now use the qemu-kvm stable branch (0.11.1)
new virtual packages for better kernel dependency handling - proxmox-ve-2.6.18, proxmox-ve-2.6.24 and proxmox-ve-2.6.32
updated most appliances
installation CD now use/install proxmox-ve-2.6.18
installer uses UUID for /boot partition
updated vzctl, with improved init-logger patch
updated Russian and Japanese translation
many other small improvements and bug fixes

Proxmox VE 1.4

Released 19.10.2009: See Downloads

Including:

First release with new storage model
iSCSI support
NFS support
LVM (managed storage, shared LVM devices like FC)
DRBD storage replication with active/active mode
Multiple ISO storages (NFS)
Multiple backup locactions (including NFS)
Zero downtime live migration (KVM); over ssh channel - all traffic is encrypted
Updated web interface (pve-manager)
Support new storage model
Allow to specify MAC address (for KVM and OpenVZ in veth mode)
Added iSCSI Tape support, see https://scst.sourceforge.net/
Added SCST modules to the kernel
Provide SCST userspace package
Updated kernel; update broadcom bnx2 and tg3 drivers; updated ARCEA RAID (acrmsr) drivers; updated kvm modules (2.6.30.1); update realtek network drivers; update intel igb driver (igb-1.3.28.4.tar.gz); update intel e1000e driver (e1000e-1.0.2.5.tar.gz); cciss driver updates; ich10 updates for Nehalem (corei7) based systems
Updated kvm modules (2.6.30.1)
Update DRBD to 8.3.2
New KVM userspace tools
New feature: assign Sockets/Cores to KVM guests
We now use the qemu-kvm stable branch (0.11.0)
Rename package pve-kvm to pve-qemu-kvm
Qemu-server updates
Support up to 1000 vlans
New migrate_speed and migrate_downtime settings
New VZDump - complete rewrite to support the new storage architecture; do not waste space in 'stop' mode
Slightly changed naming of backup files; added date and time to file name; added VM type (Openvz/Qemu) to file name; new plugin architecture; support hook scripts

Proxmox VE 1.4 beta2

Released 29.09.2009: See Downloads

Including:

Updated vzdump web interface with support new storage model
Updated kernel: update broadcom bnx2 and tg3 drivers, updated kvm modules (2.6.30.1), updated realtek network drivers, more ich10 updates for Nehalem (corei7) based systems
New kvm userspace tools
New feature: assign Sockets/Cores to KVM guests
We now use the qemu-kvm stable branch (0.11.0)
Rename package pve-kvm to pve-qemu-kvm
Qemu-server updates: support up to 1000 vlans
New migrate_speed and migrate_downtime settings

Note: the sources.list does not point to the pvetest repo, so please adapt this accordingly if you want to update/install packages from pvetest

Proxmox VE 1.4 beta1

Released 11.09.2009: See Downloads

Including:

First release with new storage model
iSCSI support
NFS support
LVM (managed storage, shared LVM devices)
DRBD storage replication with active/active mode (DRBD 8.3.2)
Multiple ISO storages (NFS)
Zero downtime live migration (KVM) over ssh channel - all traffic is encrypted
Updated kernel:update Intel igb driver (igb-1.3.28.4.tar.gz), Intel e1000e driver (e1000e-1.0.2.5.tar.gz),cciss, ich10 updates, update DRBD to 8.3.2
new VZDump: complete rewrite to support the new storage architecture; do not waste space in 'stop' mode; new plugin architecture

Note: the sources.list does not point to the pvetest repo, so please adapt this accordingly if you want to update/install packages from pvetest

Proxmox VE 1.3

Released 4.06.2009: See Downloads

Including:

Updated Kernel
Support for Highpoint RR3120
Removed OpenVZ limit of 50 machines
Update to kvm-86
Vzdump: minor bug fixes
Qemu-server: added new 'tablet' option

Proxmox VE 1.2

Released 12.05.2009: See Downloads

Including:

Based on new Debian Lenny (Debian 5.0)
New KVM with many improvements (kvm-85)
New Kernel (still 2.6.24 based but with a lot of backported drivers)
Update to aoe6-71
Update to drbd-8.3.1
Include HighPoint RocketRAID 3xxx/4xxx Controller Driver
Update to busybox 1.14.0
Use busybox mdev as firmware loader
Compiled with gcc 4.3.2 (default on Debian Lenny)
Load virtio_blk (if you run Proxmox VE inside Proxmox VE as a KVM guest)
New OpenVZ vzctl (mainly bug fixes)
Vncterm: better terminal emulation
Everything updated to Debian Lenny
Many bug fixes

Proxmox VE 1.1

Released 15.01.2009: See Downloads

Including:

Updated Kernel - new e1000/e1000e drivers
New KVM with many improvements (KVM-83)
New Chinese translations
Minor pve-manager updates and bug fixes

Proxmox VE 1.0 - First stable release

Released 29.10.2008: See Downloads

Including:

KVM and OpenVZ live migration
Web interface for vzdump backups
Improved vzdump, now support KVM guests
New templates available
Support multiple languages (see Translations for details)
Improved installer (use grub as boot loader, automatic country/keyboard detection)
Support up to 16 CPUs
Based on Debian 4.0r5
KVM updates (kvm-75)
OpenVZ updates, Fedora support added
Improved Kernel (includes drivers for ISCSI, AoE 6-64, DRBD 8.2.6, qlogic firmware for ISCSI/FC HBAs)
Many bugfixes

Proxmox VE 0.9beta2

Released 22.7.2008, see Proxmox VE 0.9beta2 released!

Including:

Extend pre-built Virtual Appliances downloads
Multi-bridge support
Bonding
VLAN support
Extend web interface for KVM guests (add hard drives, network cards, etc.)
Many bug fixes

Proxmox VE 0.9

First public release: 15.4.2008

[linbit-drbd-gh-1] 1.0 ^1.1 https://github.com/LINBIT/linstor-proxmox

[footnotes 7.0 1]