View source for NVIDIA vGPU on Proxmox VE

== Introduction ==

NVIDIA vGPU technology enables multiple virtual machines to use a single supported<ref name="supported-gpus">NVIDIA GPUs supported by vGPU https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html</ref> physical GPU.

This article explains how to use NVIDIA vGPU on Proxmox VE. The instructions were tested using an RTX A5000.

== Disclaimer ==

At the time of writing, Proxmox VE is not an officially supported platform for NVIDIA vGPU. This means that even with valid vGPU licenses, you may not be eligible for NVIDIA enterprise support for this use-case.
However, Proxmox VE's kernel is derived from the Ubuntu kernel, which is a supported platform for NVIDIA vGPU as of 2024.

Note that although we are using some consumer hardware in this article, for optimal performance in production workloads, we recommend using appropriate enterprise-grade hardware.
Please refer to NVIDIA's support page to verify hardware compatibility
<ref>NVIDIA vGPU Certified Servers https://www.nvidia.com/en-us/data-center/resources/vgpu-certified-servers/</ref>
<ref name="supported-gpus">NVIDIA GPUs supported by vGPU https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html</ref>.

== Hardware Setup ==

We're using the following hardware configuration for our test:

{| class="wikitable"
|+ Test System
|-
| CPU || Intel Core i7-12700K
|-
| Motherboard || ASUS PRIME Z690-A
|-
| Memory || 128 GB DDR5 Memory: 4x Crucial CT32G48C40U5
|-
| GPU || PNY NVIDIA RTX A5000
|}

Some NVIDIA GPUs do not have vGPU enabled by default, even though they support vGPU, like the RTX A5000 we tested. To enable vGPU there, switch the display using the NVIDIA Display Mode Selector Tool<ref>NVIDIA Display Mode Selector Tool https://developer.nvidia.com/displaymodeselector</ref>.
This will disable the display ports.

For a list of GPUs where this is necessary check their documentation<ref>Latest NVIDIA vGPU user guide: Switching the Mode of a GPU that Supports Multiple Display Modes https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#displaymodeselector</ref>.

The installation was tested on the following versions of Proxmox VE, Linux kernel, and NVIDIA drivers:

{| class="wikitable"
|-
! pve-manager !! kernel !! vGPU Software Branch !! NVIDIA Host drivers
|-
| 7.2-7 || 5.15.39-2-pve || 14.1 || 510.73.06
|-
| 7.2-7 || 5.15.39-2-pve || 14.2 || 510.85.03
|-
| 7.4-3 || 5.15.107-2-pve || 15.2 || 525.105.14
|-
| 7.4-17 || 6.2.16-20-bpo11-pve || 16.0 || 535.54.06
|-
| 8.1.4 || 6.5.11-8-pve || 16.3 || 535.154.02
|-
| 8.1.4 || 6.5.13-1-pve || 16.3 || 535.154.02
|}

'''Note''': As of July 2024, the driver technically works with 6.8 based kernels (GRID version 17.3), but the lower level interface changed so passing through a vGPU does not work like described here. For more info see https://forum.proxmox.com/threads/150840/

It is recommended to use the latest stable and supported version of Proxmox VE and NVIDIA drivers.
However, newer versions in one vGPU Software Branch should also work for the same or older kernel version.

Since version 16.0, certain cards are no longer supported by the NVIDIA vGPU driver, but are supported by the Enterprise AI driver
<ref name="supported-gpus">NVIDIA GPUs supported by vGPU https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html</ref>
<ref name="supported-gpus-ai">NVIDIA GPUs supported by AI Enterprise https://docs.nvidia.com/ai-enterprise/latest/product-support-matrix/index.html</ref>.
We have tested the Enterprise AI driver with an A16 and vGPU technology and found that it behaves similarly to the old vGPU driver. Therefore, the following steps also apply.

== Preparation ==

Before actually installing the host drivers, there are a few steps to be done on the Proxmox VE host.

'''Tip''': If you need to use a root shell, you can, for example, open one by connecting via SSH or using the node shell on the Proxmox VE web interface.

=== Enable PCIe Passthrough ===

Make sure that your system is compatible with PCIe passthrough. See the [https://pve.proxmox.com/wiki/PCI(e)_Passthrough PCI(e) Passthrough] documentation.

Additionally, confirm that the following features are enabled in your firmware settings (BIOS/UEFI):

* VT-d for Intel, or AMD-v for AMD (sometimes named IOMMU)
* SR-IOV (this may not be necessary for older pre-Ampere GPU generations)
* Above 4G decoding
* PCI AER (Advanced Error Reporting)
* PCI ASPM (Active State Power Management)

The firmware of your host might use different naming. If you are unable to locate some of these options, refer to the documentation provided by your firmware or motherboard manufacturer.

'''Note''': It is crucial to ensure that both the IOMMU options are enabled in your firmware and the kernel.

=== Setup Proxmox VE Repositories ===

Proxmox VE's comes with the enterprise repository set up by default as this repository provides better tested software and is recommended for production use.
The enterprise repository needs a valid subscription per node. For evaluation or non-production use cases you can simply switch to the public <code>no-subscription</code> repository. This provides the same feature-set but with a higher frequency of updates.

You can use the <code>Repositories</code> management panel in the Proxmox VE web UI for managing package repositories, see the [https://pve.proxmox.com/wiki/Package_Repositories documentation] for details.

=== Update to Latest Package Versions ===

Proxmox VE uses a rolling release model and should be updated frequently to ensure that your Proxmox VE installation has the latest bug and security fixes, and features available.

You can update your Proxmox VE node using the <code>update</code> panel on the web UI.

=== Blacklist the Nouveau Driver  ===

Next, you want to blacklist the open source <code>nouveau</code> kernel module to avoid it from interfering with the one from NVIDIA.

To do that, add a line with <code>blacklist nouveau</code> to a file in the <code>/etc/modprobe.d/</code> directory.
For example, open a root shell and execute:
 echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

Then, [https://pve.proxmox.com/wiki/PCI(e)_Passthrough#qm_pci_passthrough_update_initramfs update your initramfs], to ensure that the module is blocked from loading at early boot, and then reboot your host.

=== Setup DKMS ===

Because the NVIDIA module is separate from the kernel, it must be rebuilt with Dynamic Kernel Module Support (DKMS) for each new kernel update.

To set up DKMS, you must install the headers package for the kernel and the DKMS helper package. In a root shell, run

 apt update
 apt install dkms libc6-dev proxmox-default-headers --no-install-recommends

'''Note''': If you do not have the default kernel version installed, but for example an opt-in kernel, you must install the appropriate <code>proxmox-headers-X.Y</code> package instead of <code>proxmox-default-headers</code>.

== Host Driver Installation ==

'''Note''': The driver/file versions shown in this section are examples only; use the correct file names for the selected driver you're installing.

To get started, you will need the appropriate host and guest drivers; see the NVIDIA Virtual GPU Software Quick Start Guide<ref>Getting your NVIDIA GRID Software: https://docs.nvidia.com/grid/latest/grid-software-quick-start-guide/index.html#getting-your-nvidia-grid-software</ref> for instructions on how to obtain them.
Choose <code>Linux KVM</code> as target hypervisor when downloading.

In our case we got the following host driver file:
 NVIDIA-Linux-x86_64-525.105.14-vgpu-kvm.run
Copy this file over to your Proxmox VE node.

To start the installation, you need to make the installer executable first, and then pass the <code>--dkms</code> option when running it, to ensure that the module is rebuilt after a kernel upgrade:
 chmod +x NVIDIA-Linux-x86_64-525.105.14-vgpu-kvm.run
 ./NVIDIA-Linux-x86_64-525.105.14-vgpu-kvm.run --dkms
Follow the steps of the installer.

After the installer has finished successfully, you will need to reboot your system, either using the web interface or by executing <code>reboot</code>.

=== Enabling SR-IOV ===

On some NVIDIA GPUs (for example, those based on the Ampere architecture), you must first enable SR-IOV before being able to use vGPUs.
You can do that with the <code>sriov-manage</code> script from NVIDIA.

 /usr/lib/nvidia/sriov-manage -e <pciid|ALL>
<!-- TODO: describe what <pci-id|ALL> is and why not always all, ...  -->

Since that setting gets lost on reboot, it might be a good idea to write a cronjob or systemd service to enable it on reboot.

Here is an example systemd service for enabling SR-IOV on all found NVIDIA GPUs:

<pre>
[Unit]
Description=Enable NVIDIA SR-IOV
After=network.target nvidia-vgpud.service nvidia-vgpu-mgr.service
Before=pve-guests.service

[Service]
Type=oneshot
ExecStart=/usr/lib/nvidia/sriov-manage -e ALL

[Install]
WantedBy=multi-user.target
</pre>

Depending on the actual hardware, it might be necessary to give the <code>nvidia-vgpud.service</code> a bit more time to start, you can do that by adding
 ExecStartPre=/bin/sleep 5
just before the <code>ExecStart</code> line in the service file (replace '5' by an appropriate amount of seconds).

You can save this in <code>/usr/local/lib/systemd/system/nvidia-sriov.service</code>. Then enable and start it with:

 systemctl daemon-reload
 systemctl enable --now nvidia-sriov.service

This will then run after the <code>nvidia-daemons</code> got started, but before the Proxmox VE virtual guest auto start-up.

Verify that there are multiple virtual functions for your device with:

 # lspci -d 10de:

In our case there are now 24 virtual functions in addition to the physical card (01:00.0):

 01:00.0 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:00.4 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:00.5 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:00.6 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:00.7 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.0 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.1 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.2 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.3 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.4 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.5 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.6 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:01.7 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.0 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.1 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.2 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.3 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.4 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.5 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.6 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:02.7 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:03.0 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:03.1 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:03.2 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)
 01:03.3 3D controller: NVIDIA Corporation GA102GL [RTX A5000] (rev a1)

== Guest Configuration ==

=== General Setup ===

First, set up a VM as you normally would, without adding a vGPU.

After configuring the VM to your liking, shut down the VM and add a vGPU by selecting one of the virtual functions and selecting the appropriate mediated device type.

For example:

Via the CLI:

 qm set VMID -hostpci0 01:00.4,mdev=nvidia-660

Via the web interface:

[[File:Pve select vgpu.png|500px|none|Selecting a vGPU model]]

To find the correct mediated device type, you can use <code>sysfs</code>.
Here is a sample shell script that prints the type, then the name (which corresponds to the NVIDIA documentation) and the description, which contains helpful information (such as the maximum number of instances available).
Adjust the PCI path to your needs:

<pre>
#!/bin/sh
set -e

for i in /sys/bus/pci/devices/0000:01:00.4/mdev_supported_types/*; do
    basename "$i"
    cat "$i/name"
    cat "$i/description"
    echo
done
</pre>

Since pve-manager version 7.2-8 and libpve-common-perl version 7.2-3, the GUI shows the correct name for the type.

If your qemu-server version is below 7.2-4, you must add an additional parameter
to the vm:

 # qm set VMID -args '-uuid <UUID-OF-THE-MDEV>'

The UUID of the mediated device is automatically generated from the VMID and the <code>hostpciX</code> index of the config, where the host PCI index is used as the first part and the VMID as the last part.
For example, if you configure <code>hostpci2</code> for VM with VMID <code>12345</code>, the generated UUID will be

 00000002-0000-0000-0000-000000012345

You can now start the VM and continue configuring the guest from within.

We tested a Windows 10 and Ubuntu 22.04 installation, but the setup will be similar for other supported operating systems.

=== Windows 10 ===

First install and configure a desktop sharing software that matches your requirements. Some
examples of such software include:

* '''VNC'''<br />many different options, some free, some commercial
* '''Remote Desktop'''<br />built into Windows itself
* '''Parsec'''<br />Costs money for commercial use, allows using hardware accelerated encoding
* '''RustDesk'''<br />free and open source, but relatively new as of 2022

We used simple Windows built-in remote desktop for testing.

[[File:Windows rdp.png|thumb|Enabling Remote Desktop in Windows 10]]

Then you can install the Windows guest driver that is published by NVIDIA.
Refer to their documentation<ref name="gpu-mapping">NVIDIA Virtual GPU (vGPU) Software Documentation https://docs.nvidia.com/grid/</ref>to find a compatible guest driver to host driver mapping.
In our case this was the file

 528.89_grid_win10_win11_server2019_server2022_dch_64bit_international.exe

Start the installer and follow the instructions, then, after it finished restart the guest as prompted.

<gallery>
Windows nv install01.png|Starting NVIDIA driver installation
Windows nv install02.png|Accepting the license agreement
Windows nv install03.png|Finishing the installation
</gallery>

From this point on, the integrated noVNC console of PVE will not be usable anymore, so use
your desktop sharing software to connect to the guest. Now you can use the vGPU for starting
3D applications such as Blender, 3D games, etc.

<gallery>
Windows valley.png|Unigine Valley
Windows supertuxkart.png|SuperTuxKart
Windows blender.png|Blender
</gallery>

=== Ubuntu 22.04 Desktop ===

Before installing the guest driver, install and configure a desktop sharing software, for example:

* '''VNC'''<br />many options. We use x11vnc here, which is free and open source, but does not currently provide hardware accelerated encoding
* '''NoMachine'''</br>provides hardware accelerated encoding, but is not open source and costs money for business use
* '''RustDesk'''<br />free and open source, but relatively new as of 2022

We installed x11vnc in this example. While we're showing how to install and configure it, this is not the only way to achieve the goal of having properly configured desktop sharing.

Since Ubuntu 22.04 ships GDM3 + Gnome + Wayland per default, you first need to switch the login manager to one that uses X.org.
We successfully tested LightDM here, but others may work as well.

 # apt install lightdm

Select 'LightDM' as default login manager when prompted. After that install <code>x11vnc</code> with

 # apt install x11vnc

We then added a systemd service that starts the VNC server on the x.org server provided by LightDM in <code>/etc/systemd/system/x11vnc.service</code>

<pre>
[Unit]
Description=Start x11vnc
After=multi-user.target

[Service]
Type=simple
ExecStart=/usr/bin/x11vnc -display :0 -auth /var/run/lightdm/root/:0 -forever -loop -repeat -rfbauth /etc/x11vnc.passwd -rfbport 5900 -shared -noxdamage

[Install]
WantedBy=multi-user.target
</pre>

You can set the password by executing:

 # x11vnc -storepasswd /etc/x11vnc.passwd
 # chmod 0400 /etc/x11vnc.passwd

After setting up LightDM and x11vnc and restarting the VM, you can connect via VNC.

Now, install the .deb package that NVIDIA provides for Ubuntu.
Check the NVIDIA documentation<ref name="gpu-mapping" /> for a compatible guest driver to host driver mapping.

In our case this was <code>nvidia-linux-grid-525_525.105.17_amd64.deb</code>, and we directly installed from the local file using apt.
For that to work you must prefix the relative path, for example <code>./</code> if the <code>.deb</code> file is located in the current directory.

 # apt install ./nvidia-linux-grid-525_525.105.17_amd64.deb

Then you must use NVIDIA's tools to configure the x.org configuration with:

 # nvidia-xconfig

Now you can reboot and use a VNC client to connect and use the vGPU for 3D applications.

<gallery>
Ubuntu valley.png|Unigine Valley
Nv Ubuntu supertuxkart.png|SuperTuxKart
Ubuntu blender.png|Blender
</gallery>

{{Note| If you want to use CUDA on a Linux Guest, you must install the CUDA Toolkit manually<ref>NVIDIA CUDA Toolkit Download https://developer.nvidia.com/cuda-downloads</ref>.
Check the NVIDIA documentation which version of CUDA is supported for your vGPU drivers.

In our case we needed to install CUDA 11.6 (only the toolkit, not the driver) with the file:

 cuda_11.6.2_510.47.03_linux.run 
|warn}}

=== Guest vGPU Licensing ===

To use the vGPU without restriction, you must adhere to NVIDIA's licensing.
Check the NVIDIA vGPU documentation<ref>NVIDIA GRID Licensing User Guide: https://docs.nvidia.com/grid/latest/grid-licensing-user-guide/index.html</ref> for instructions on how to do so.

'''Tip''': Ensure that the guest system time is properly synchronized using NTP, otherwise the guest will be unable to request a license for the vGPU.

== Notes ==
<references />

[[Category: HOWTO]]
[[Category: Qemu/KVM]]