Network Model

From Proxmox VE
Jump to: navigation, search

Proxmox VE uses a bridged networking model. Each host can have up to 4094 bridges. Bridges are like physical network switches implemented in software on the Proxmox VE host. All VMs can share one bridge as if virtual network cables from each guest were all plugged into the same switch. For connecting VMs to the outside world, bridges are attached to physical network cards assigned a TCP/IP configuration. For further flexibility, VLANs (IEEE 802.1q) and network bonding/aggregation are possible. In this way it is possible to build complex, flexible virtual networks.

The network configuration is usually changed using the web interface. Changes are stored to /etc/network/interfaces.new, and are activated when you reboot the host. Actual configuration resides in /etc/network/interfaces. The following examples list the contents of that file.

Default Configuration (bridged)

The installation program creates a single bridge (vmbr0), which is connected to the first ethernet card (eth0).

auto lo
iface lo inet loopback
iface eth0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.10.2
        netmask 255.255.255.0
        gateway 192.168.10.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

Virtual machines behave as if they were directly connected to the physical network. The network, in turn, sees each virtual machine as having its own MAC, even though there is only one network cable connecting all of these VMs to the network.

Routed Configuration

Most hosting providers do not support the above setup. For security reasons they disable networking as soon as they detect multiple MAC addresses on a single interface. See discussion on multiple subnets on proxmox using different gateways.

A common setup is a public IP (assume 192.168.10.2 for this example), and additional IP blocks for your VMs (10.10.10.1/255.255.255.0). For such situations we recommend the following setup:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address  192.168.10.2
        netmask  255.255.255.0
        gateway  192.168.10.1
        post-up echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp


auto vmbr0
iface vmbr0 inet static
        address  10.10.10.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

Masquerading (NAT) with iptables

In some cases you may want to use private IPs behind your Proxmox host's true IP, and masquerade the traffic using NAT:

auto lo
iface lo inet loopback

auto eth0
#real IP adress 
iface eth0 inet static
        address  192.168.10.2
        netmask  255.255.255.0
        gateway  192.168.10.1

auto vmbr0
#private sub network
iface vmbr0 inet static
        address  10.10.10.1
        netmask  255.255.255.0
        bridge_ports none
        bridge_stp off
        bridge_fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o eth0 -j MASQUERADE

QEMU port redirection

If you have your VM network configuration set to NAT mode, there's another option. According to this thread, QEMU can do port forwarding by itself for TCP and UDP only, but both the web UI and the qm(1) command current lacks the ability to control this directly. Instead, you run:

qm set <vmid> -args "--redir tcp:5555::22"

in order to, say, redirect host port tcp/5555 to guest port tcp/22. At this point you should be able to ssh to the NAT-mode guest VM by connecting to port 5555 on the host's IP.

Configuring VLAN in a cluster

For the simplest way to create VLAN follow the link: VLAN

Goal:

  • Have two separate networks on the same NIC
  • Another host (firewall) manages the routing and rules to access these VMs (out of this doc)

Suppose this scenario:

  • A cluster with two nodes
  • Each node has two NICs
  • We want to bond the NICs
  • We use two networks: one untagged 192.168.1.0/24 and one tagged (VLanID=53) 192.168.2.0/24, we must configure the switch with port vlan.
  • We want to separate these networks at layer 2

Create bond0

First of all we create the bond0 (switch assisted 802.3ad) at the proxmox web interface, follow the video.

Please note that no matter how many nics you add to a bond the speed for a single connection can never be higher than the speed of an individual nic. What bonding means is that more connections can in parallel run with the full speed of a single connection.

At the end we have a /etc/network/interface like this:

# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto bond0
iface bond0 inet manual
	slaves eth0 eth1
	bond_miimon 100
	bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
	address  192.168.1.1
	netmask  255.255.255.0
	gateway  192.168.1.250
	bridge_ports bond0
	bridge_stp off
	bridge_fd 0

Configure your switch appropriately. If you're using a bond of multiple links, you need to tell this to your switch and put the switch ports in a Link Aggregation Group or Trunk.

Create VLAN

We have two methods to follow:

First explicit method

auto vlan53
iface vlan53 inet manual
	vlan_raw_device bond0

Second method

We can use directly the NIC dot VLAN ID, like bond0.53

I prefer the first one!

Create the bridge manually

Now we create the second bridge manually. Here we assume the gateway as 192.168.2.250 for the 192.168.2.0/24 network.

auto vmbr1
iface vmbr1 inet static
	address  192.168.2.1
	netmask  255.255.255.0
	network  192.168.2.0
	bridge_ports vlan53
	bridge_stp off
	bridge_fd 0
	post-up ip route add table vlan53 default via 192.168.2.250 dev vmbr1
	post-up ip rule add from 192.168.2.0/24 table vlan53
	post-down ip route del table vlan53 default via 192.168.2.250 dev vmbr1
	post-down ip rule del from 192.168.2.0/24 table vlan53

NOTE:

  • We must not indicate the gateway, we must manually modify the routing table use ip route 2
  • The whole configuration must replicate on the other cluster's node, the only change is the node's IP.

Create the table in ip route 2

We must change the file /etc/iproute2/rt_tables, add the following line:

# Table for vlan53
53  vlan53

use these commands to add:

echo "# Table for vlan53" >> /etc/iproute2/rt_tables
echo "53 vlan53" >> /etc/iproute2/rt_tables

Create the vlan on switch

For example on a HP Procurve 52 ports we use the following instructions to create the vlan.

Suppose:

  • Ports 47-48 trunk (switch assisted 802.3ad) for gateway
  • Ports 1-2 trunk (switch assisted 802.3ad) for the first node of cluster proxmox
  • Ports 3-4 trunk (switch assisted 802.3ad) for the second node

Enter in configuration mode and type:

trunk 1-2 Trk1 LACP 
trunk 3-4 Trk2 LACP 
trunk 47-48 Trk3 LACP 
vlan 2 
   name "Vlan2" 
   untagged Trk1-Trk3
   ip address 192.168.1.254 255.255.255.0 
   exit 
vlan 53 
   name "Vlan53" 
   tagged Trk1-Trk3
   exit 

Test the configuration

Reboot the cluster node one by one to test this configuration.

Unsupported Routing

Physical NIC (eg., eth1) cannot currently be made available exclusively for a particular KVM / Container , ie., without bridge and/or bond.

Naming Conventions

  • Ethernet devices: eth0 - eth99
  • Allowable bridge names: vmbrn, where 0 ≤ n ≤ 4094
  • Bonds: bond0 - bond9
  • VLANs: Simply add the VLAN number to the ethernet device name, separated by a period. For example "eth0.50"

Video Tutorials

References