Difference between revisions of "Software Defined Network"

From Proxmox VE
Jump to navigation Jump to search
 
Line 4: Line 4:
 
[[Category:Reference Documentation]]
 
[[Category:Reference Documentation]]
 
<pvehide>
 
<pvehide>
The Software Defined Network (SDN) feature allows one to create
+
The Software Defined Network (SDN) feature allows you to create
virtual networks (vnets) at datacenter level.
+
virtual networks (VNets) at the datacenter level.
 
SDN is currently an experimental feature in Proxmox VE. This
 
SDN is currently an experimental feature in Proxmox VE. This
Documentation for it is also still under development, ask on our
+
documentation for it is also still under development. Ask on our
 
mailing lists or in the forum for questions and feedback.
 
mailing lists or in the forum for questions and feedback.
 
Installation
 
Installation
To enable the experimental SDN integration, you need to install the
+
To enable the experimental Software Defined Network (SDN) integration, you need
libpve-network-perl and ifupdown2 package on every node:
+
to install the libpve-network-perl and ifupdown2 packages on every node:
 
apt update
 
apt update
 
apt install libpve-network-perl ifupdown2
 
apt install libpve-network-perl ifupdown2
After that you need to add the following line:
+
Proxmox VE version 7 and above come installed with ifupdown2.
 +
After this, you need to add the following line to the end of the
 +
/etc/network/interfaces configuration file, so that the SDN configuration gets
 +
included and activated.
 
source /etc/network/interfaces.d/*
 
source /etc/network/interfaces.d/*
at the end of the /etc/network/interfaces configuration file, so that the SDN
 
config gets included and activated.
 
 
Basic Overview
 
Basic Overview
The Proxmox VE SDN allows separation and fine grained control of Virtual Guests
+
The Proxmox VE SDN allows for separation and fine-grained control of virtual guest
networks, using flexible software controlled configurations.
+
networks, using flexible, software-controlled configurations.
Separation consists of zones, a zone is it&#8217;s own virtual separated network area.
+
Separation is managed through zones, where a zone is its own virtual separated
A VNet is a type of a virtual network connected to a zone. Depending on which
+
network area. A VNet is a type of a virtual network connected to a zone.
type or plugin the zone uses it can behave differently and offer different
+
Depending on which type or plugin the zone uses, it can behave differently and
features, advantages or disadvantages.
+
offer different features, advantages, and disadvantages. Normally, a VNet
Normally a VNet shows up as a common Linux bridge with either a VLAN or
+
appears as a common Linux bridge with either a VLAN or VXLAN tag, however,
VXLAN tag, but some can also use layer 3 routing for control.
+
some can also use layer 3 routing for control. VNets are deployed locally on
The VNets are deployed locally on each node, after configuration was committed
+
each node, after being configured from the cluster-wide datacenter SDN
from the cluster-wide datacenter SDN administration interface.
+
administration interface.
Main configuration
+
Main Configuration
The configuration is done at datacenter (cluster-wide) level, it will be saved
+
Configuration is done at the datacenter (cluster-wide) level and is saved in
in configuration files located in the shared configuration file system:
+
files located in the shared configuration file system:
 
/etc/pve/sdn
 
/etc/pve/sdn
On the web-interface SDN feature have 3 main sections for the configuration
+
On the web-interface, SDN features 3 main sections:
SDN: a overview of the SDN state
+
SDN: An overview of the SDN state
Zones: Create and manage the virtual separated network Zones
+
Zones: Create and manage the virtually separated network zones
VNets: Create virtual network bridges + subnets management.
+
VNets: Create virtual network bridges and manage subnets
And some options:
+
In addition to this, the following options are offered:
Controller: For complex setups to control Layer 3 routing
+
Controller: For controlling layer 3 routing in complex setups
Sub-nets: Used to defined ip networks on VNets.
+
Subnets: Used to defined IP networks on VNets
IPAM: Allow to use external tools for IP address management (guest IPs)
+
IPAM: Enables the use of external tools for IP address management (guest
DNS: Allow to define a DNS server api for registering a virtual guests
+
  IPs)
  hostname and IP-addresses
+
DNS: Define a DNS server API for registering virtual guests' hostname and IP
 +
  addresses
 
SDN
 
SDN
This is the main status panel. Here you can see deployment status of zones on
+
This is the main status panel. Here you can see the deployment status of zones
different nodes.
+
on different nodes.
There is an Apply button, to push and reload local configuration on all
+
The Apply button is used to push and reload local configuration on all cluster
cluster nodes.
+
nodes.
 
Local Deployment Monitoring
 
Local Deployment Monitoring
After applying the configuration through the main SDN web-interface panel,
+
After applying the configuration through the main SDN panel,
 
the local network configuration is generated locally on each node in
 
the local network configuration is generated locally on each node in
/etc/network/interfaces.d/sdn, and with ifupdown2 reloaded.
+
the file /etc/network/interfaces.d/sdn, and reloaded with ifupdown2.
You can monitor the status of local zones and vnets through the main tree.
+
You can monitor the status of local zones and VNets through the main tree.
 
Zones
 
Zones
A zone will define a virtually separated network.
+
A zone defines a virtually separated network. Zones can be restricted to
It can use different technologies for separation:
+
specific nodes and assigned permissions, in order to restrict users to a certain
VLAN: Virtual LANs are the classic method to sub-divide a LAN
+
zone and its contained VNets.
QinQ: stacked VLAN (formally known as IEEE 802.1ad)
+
Different technologies can be used for separation:
VXLAN: (layer2 vxlan)
+
VLAN: Virtual LANs are the classic method of subdividing a LAN
Simple: Isolated Bridge, simple l3 routing bridge (NAT)
+
QinQ: Stacked VLAN (formally known as IEEE 802.1ad)
bgp-evpn: vxlan using layer3 border gateway protocol routing
+
VXLAN: Layer2 VXLAN
You can restrict a zone to specific nodes.
+
Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)
It&#8217;s also possible to add permissions on a zone, to restrict user to use only a
+
EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing
specific zone and only the VNets in that zone
 
 
Common options
 
Common options
The following options are available for all zone types.
+
The following options are available for all zone types:
 
nodes
 
nodes
Deploy and allow to use a VNets configured for this Zone only on these
+
The nodes which the zone and associated VNets should be deployed on
nodes.
 
 
ipam
 
ipam
Optional, if you want to use an ipam tool to manage ips in this zone
+
Optional. Use an IP Address Management (IPAM) tool to manage IPs in the
 +
  zone.
 
dns
 
dns
Optional, dns api server.
+
Optional. DNS API server.
 
reversedns
 
reversedns
Optional, reverse dns api server.
+
Optional. Reverse DNS API server.
 
dnszone
 
dnszone
Optional, dns domain name. Use to register hostname like
+
Optional. DNS domain name. Used to register hostnames, such as
&lt;hostname&gt;.&lt;domain&gt;. The dns zone need to be already existing in dns server.
+
  &lt;hostname&gt;.&lt;domain&gt;. The DNS zone must already exist on the DNS server.
 
Simple Zones
 
Simple Zones
This is the simplest plugin, it will create an isolated vnet bridge.
+
This is the simplest plugin. It will create an isolated VNet bridge.
This bridge is not linked to physical interfaces, VM traffic is only
+
This bridge is not linked to a physical interface, and VM traffic is only
local to the node(s).
+
local between the node(s).
It can be also used for NAT or routed setup.
+
It can also be used in NAT or routed setups.
 
VLAN Zones
 
VLAN Zones
This plugin will reuse an existing local Linux or OVS bridge,
+
This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs
and manage VLANs on it.
+
on it. The benefit of using the SDN module is that you can create different
The benefit of using SDN module, is that you can create different zones with
+
zones with specific VNet VLAN tags, and restrict virtual machines to separated
specific VNets VLAN tag, and restrict Virtual Machines to separated zones.
+
zones.
 
Specific VLAN configuration options:
 
Specific VLAN configuration options:
 
bridge
 
bridge
Reuse this local bridge or OVS switch, already
+
Reuse this local bridge or OVS switch, already configured on each
configured on each local node.
+
  local node.
 
QinQ Zones
 
QinQ Zones
QinQ is stacked VLAN. The first VLAN tag defined for the zone
+
QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the
(so called service-vlan), and the second VLAN tag defined for the vnets
+
zone (the service-vlan), and the second VLAN tag is defined for the
Your physical network switches must support stacked VLANs!
+
VNets.
Specific QinQ configuration options:
+
Your physical network switches must support stacked VLANs for this
 +
configuration!
 +
Below are the configuration options specific to QinQ:
 
bridge
 
bridge
A local VLAN-aware bridge already configured on each local node
+
A local, VLAN-aware bridge that is already configured on each local
 +
  node
 
service vlan
 
service vlan
 
The main VLAN tag of this zone
 
The main VLAN tag of this zone
 
service vlan protocol
 
service vlan protocol
allow to define a 802.1q (default) or 802.1ad service vlan type.
+
Allows you to choose between an 802.1q (default) or
 +
  802.1ad service VLAN type.
 
mtu
 
mtu
Due to the double stacking of tags you need 4 more bytes for QinQ VLANs.
+
Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs.
For example, you reduce the MTU to 1496 if you physical interface MTU is
+
  For example, you must reduce the MTU to 1496 if you physical interface MTU is
1500.
+
  1500.
 
VXLAN Zones
 
VXLAN Zones
The VXLAN plugin will establish a tunnel (named overlay) on top of an existing
+
The VXLAN plugin establishes a tunnel (overlay) on top of an existing
network (named underlay). It encapsulate layer 2 Ethernet frames within layer
+
network (underlay). This encapsulates layer 2 Ethernet frames within layer
 
4 UDP datagrams, using 4789 as the default destination port. You can, for
 
4 UDP datagrams, using 4789 as the default destination port. You can, for
 
example, create a private IPv4 VXLAN network on top of public internet network
 
example, create a private IPv4 VXLAN network on top of public internet network
 
nodes.
 
nodes.
This is a layer2 tunnel only, no routing between different VNets is possible.
+
This is a layer 2 tunnel only, so no routing between different VNets is
Each VNet will have use specific VXLAN id from the range (1 - 16777215).
+
possible.
 +
Each VNet will have a specific VXLAN ID in the range 1 - 16777215.
 
Specific EVPN configuration options:
 
Specific EVPN configuration options:
 
peers address list
 
peers address list
A list of IPs from all nodes through which you want to
+
A list of IP addresses from each node through which you
communicate. Can also be external nodes.
+
  want to communicate. Can also be external nodes.
 
mtu
 
mtu
Because VXLAN encapsulation use 50bytes, the MTU need to be 50 bytes
+
Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
lower than the outgoing physical interface.
+
  lower than the outgoing physical interface.
 
EVPN Zones
 
EVPN Zones
This is the most complex of all supported plugins.
+
This is the most complex of all the supported plugins.
BGP-EVPN allows one to create routable layer3 network. The VNet of EVPN can
+
BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can
have an anycast IP-address and or MAC-address. The bridge IP is the same on each
+
have an anycast IP address and/or MAC address. The bridge IP is the same on each
node, with this a virtual guest can use that address as gateway.
+
node, meaning a virtual guest can use this address as gateway.
 
Routing can work across VNets from different zones through a VRF (Virtual
 
Routing can work across VNets from different zones through a VRF (Virtual
 
Routing and Forwarding) interface.
 
Routing and Forwarding) interface.
Specific EVPN configuration options:
+
The configuration options specific to EVPN are as follows:
 
VRF VXLAN tag
 
VRF VXLAN tag
This is a vxlan-id used for routing interconnect between vnets,
+
This is a VXLAN-ID used for routing interconnect between VNets.
it must be different than VXLAN-id of VNets
+
  It must be different than the VXLAN-ID of the VNets.
 
controller
 
controller
an EVPN-controller need to be defined first (see controller
+
An EVPN-controller must to be defined first (see controller plugins
plugins section)
+
  section).
 
VNet MAC address
 
VNet MAC address
A unique anycast MAC address for all VNets in this zone.
+
A unique, anycast MAC address for all VNets in this zone.
 
   Will be auto-generated if not defined.
 
   Will be auto-generated if not defined.
 
Exit Nodes
 
Exit Nodes
This is used if you want to define some proxmox nodes, as exit
+
Optional. This is used if you want to define some Proxmox VE nodes as
   gateway from evpn network through real network. The configured nodes will
+
   exit gateways from the EVPN network, through the real network. The configured
  announce a default route in the EVPN network.
+
  nodes will announce a default route in the EVPN network.
Advertise Subnets
+
Primary Exit Node
Optional. If you have silent vms/CT (for example, multiples
+
Optional. If you use multiple exit nodes, this forces
   ips by interfaces, and the anycast gateway don&#8217;t see traffic from theses ips,
+
   traffic to a primary exit node, instead of load-balancing on all nodes. This
   the ips addresses won&#8217;t be able to be reach inside the evpn network). This
+
   is required if you want to use SNAT or if your upstream router doesn&#8217;t support
   option will announce the full subnet in the evpn network in this case.
+
   ECMP.
 
Exit Nodes local routing
 
Exit Nodes local routing
 
Optional. This is a special option if you need to
 
Optional. This is a special option if you need to
   reach a vm/ct service from an exit node. (By default, the exit nodes only
+
   reach a VM/CT service from an exit node. (By default, the exit nodes only
   allow forwarding traffic between real network and evpn network).
+
   allow forwarding traffic between real network and EVPN network).
 +
Advertise Subnets
 +
Optional. If you have silent VMs/CTs (for example, if you
 +
  have multiple IPs and the anycast gateway doesn&#8217;t see traffic from theses IPs,
 +
  the IP addresses won&#8217;t be able to be reach inside the EVPN network). This
 +
  option will announce the full subnet in the EVPN network in this case.
 +
Disable Arp-Nd Suppression
 +
Optional. Don&#8217;t suppress ARP or ND packets.
 +
  This is required if you use floating IPs in your guest VMs
 +
  (IP are MAC addresses are being moved between systems).
 +
Route-target import
 +
Optional. Allows you to import a list of external EVPN
 +
  route targets. Used for cross-DC or different EVPN network interconnects.
 
MTU
 
MTU
because VXLAN encapsulation use 50 bytes, the MTU needs to be 50 bytes
+
Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes
   lower than the maximal MTU of the outgoing physical interface.
+
   less than the maximal MTU of the outgoing physical interface.
 
VNets
 
VNets
A VNet is in its basic form just a Linux bridge that will be deployed locally
+
A VNet is, in its basic form, a Linux bridge that will be deployed locally on
on the node and used for Virtual Machine communication.
+
the node and used for virtual machine communication.
VNet properties are:
+
The VNet configuration properties are:
 
ID
 
ID
a 8 characters ID to name and identify a VNet
+
An 8 character ID to name and identify a VNet
 
Alias
 
Alias
 
Optional longer name, if the ID isn&#8217;t enough
 
Optional longer name, if the ID isn&#8217;t enough
Line 166: Line 184:
 
The associated zone for this VNet
 
The associated zone for this VNet
 
Tag
 
Tag
The unique VLAN or VXLAN id
+
The unique VLAN or VXLAN ID
 
VLAN Aware
 
VLAN Aware
Allow to add an extra VLAN tag in the virtual machine or
+
Enable adding an extra VLAN tag in the virtual machine or
  container vNIC configurations or allow the guest OS to manage the VLAN&#8217;s tag.
+
container&#8217;s vNIC configuration, to allow the guest OS to manage the VLAN&#8217;s tag.
Sub-Nets
+
Subnets
A sub-network (subnet or sub-net) allows you to define a specific IP network
+
A subnetwork (subnet) allows you to define a specific IP network
(IPv4 or IPv6). For each VNET, you can define one or more subnets.
+
(IPv4 or IPv6). For each VNet, you can define one or more subnets.
 
A subnet can be used to:
 
A subnet can be used to:
restrict IP-addresses you can define on a specific VNET
+
Restrict the IP addresses you can define on a specific VNet
assign routes/gateway on a VNET in layer 3 zones
+
Assign routes/gateways on a VNet in layer 3 zones
enable SNAT on a VNET in layer 3 zones
+
Enable SNAT on a VNet in layer 3 zones
auto assign IPs on virtual guests (VM or CT) through IPAM plugin
+
Auto assign IPs on virtual guests (VM or CT) through IPAM plugins
 
DNS registration through DNS plugins
 
DNS registration through DNS plugins
If an IPAM server is associated to the subnet zone, the subnet prefix will be
+
If an IPAM server is associated with the subnet zone, the subnet prefix will be
 
automatically registered in the IPAM.
 
automatically registered in the IPAM.
 
Subnet properties are:
 
Subnet properties are:
 
ID
 
ID
a cidr network address. Ex: 10.0.0.0/8
+
A CIDR network address, for example 10.0.0.0/8
 
Gateway
 
Gateway
ip address for the default gateway of the network.
+
The IP address of the network&#8217;s default gateway. On layer 3 zones
          On layer3 zones (simple/evpn plugins), it&#8217;ll be deployed on the vnet.
+
  (Simple/EVPN plugins), it will be deployed on the VNet.
Snat
+
SNAT
Optional, Enable Snat for layer3 zones (simple/evpn plugins) for this subnet.
+
Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this
      The subnet source ip will be natted to server outgoing interface/ip.
+
  subnet. The subnet&#8217;s source IP will be NATted to server&#8217;s outgoing interface/IP.
      On evpn zone, it&#8217;s done only on evpn gateway-nodes.
+
  On EVPN zones, this is only done on EVPN gateway-nodes.
 
Dnszoneprefix
 
Dnszoneprefix
Optional, add a prefix to domain registration, like &lt;hostname&gt;.prefix.&lt;domain&gt;
+
Optional. Add a prefix to the domain registration, like
 +
&lt;hostname&gt;.prefix.&lt;domain&gt;
 
Controllers
 
Controllers
 
Some zone types need an external controller to manage the VNet control-plane.
 
Some zone types need an external controller to manage the VNet control-plane.
Line 203: Line 222:
 
Configuration options:
 
Configuration options:
 
asn
 
asn
A unique BGP ASN number. It&#8217;s highly recommended to use private ASN
+
A unique BGP ASN number. It&#8217;s highly recommended to use a private ASN
number (64512 – 65534, 4200000000 – 4294967294), as else you could end up
+
  number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up
breaking, or get broken, by global routing by mistake.
+
  breaking global routing by mistake.
 
peers
 
peers
An ip list of all nodes where you want to communicate for the EVPN (could be also
+
An IP list of all nodes where you want to communicate for the EVPN
external nodes or route reflectors servers)
+
  (could also be external nodes or route reflectors servers)
 
BGP Controller
 
BGP Controller
The bgp controller is not used directly by a zone.
+
The BGP controller is not used directly by a zone.
You can used it to configure frr to manage bgp peers.
+
You can use it to configure FRR to manage BGP peers.
For BGP-evpn, it can be use to define a different ASN by node, so doing EBGP.
+
For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.
 
Configuration options:
 
Configuration options:
 
node
 
node
 
The node of this BGP controller
 
The node of this BGP controller
 
asn
 
asn
A unique BGP ASN number. It&#8217;s highly recommended to use private ASN
+
A unique BGP ASN number. It&#8217;s highly recommended to use a private ASN
   number from the range (64512 - 65534) or (4200000000 - 4294967294), as else
+
   number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise
   you could end up breaking, or get broken, by global routing by mistake.
+
   you could break global routing by mistake.
 
peers
 
peers
An IP list of peers you want to communicate with for the underlying
+
A list of peer IP addresses you want to communicate with using the
   BGP network.
+
   underlying BGP network.
 
ebgp
 
ebgp
If your peer&#8217;s remote-AS is different, it&#8217;s enabling EBGP.
+
If your peer&#8217;s remote-AS is different, this enables EBGP.
 
loopback
 
loopback
If you want to use a loopback or dummy interface as source for the
+
Use a loopback or dummy interface as the source of the EVPN network
   evpn network. (for multipath)
+
   (for multipath).
 
ebgp-mutltihop
 
ebgp-mutltihop
if the peers are not directly connected or use loopback, you can increase the
+
Increase the number of hops to reach peers, in case they are
  number of hops to reach them.
+
  not directly connected or they use loopback.
 +
bgp-multipath-as-path-relax
 +
Allow ECMP if your peers have different ASN.
 
IPAMs
 
IPAMs
IPAM (IP address management) tools, are used to manage/assign ips on your devices on the network.
+
IPAM (IP Address Management) tools are used to manage/assign the IP addresses of
It can be used to find free ip address when you create a vm/ct for example (not yet implemented).
+
guests on the network. It can be used to find free IP addresses when you create
An IPAM is associated to 1 or multiple zones, to provide ip addresses for all subnets defined in this zone.
+
a VM/CT for example (not yet implemented).
Proxmox VE IPAM plugin
+
An IPAM can be associated with one or more zones, to provide IP addresses
This is the default internal IPAM for your proxmox cluster if you don&#8217;t have
+
for all subnets defined in those zones.
external ipam software
+
Proxmox VE IPAM Plugin
phpIPAM plugin
+
This is the default internal IPAM for your Proxmox VE cluster, if you don&#8217;t have
 +
external IPAM software.
 +
phpIPAM Plugin
 
https://phpipam.net/
 
https://phpipam.net/
You need to create an application in phpipam, and add an api token with admin
+
You need to create an application in phpIPAM and add an API token with admin
permission
+
privileges.
phpIPAM properties are:
+
The phpIPAM configuration properties are:
 
url
 
url
 
The REST-API endpoint: http://phpipam.domain.com/api/&lt;appname&gt;/
 
The REST-API endpoint: http://phpipam.domain.com/api/&lt;appname&gt;/
Line 248: Line 271:
 
An API access token
 
An API access token
 
section
 
section
An integer ID. Sections are group of subnets in phpIPAM. Default
+
An integer ID. Sections are a group of subnets in phpIPAM. Default
installations use sectionid=1 for customers.
+
  installations use sectionid=1 for customers.
Netbox IPAM plugin
+
NetBox IPAM Plugin
NetBox is an IP address management (IPAM) and data center infrastructure
+
NetBox is an IP address management (IPAM) and datacenter infrastructure
management (DCIM) tool, see the source code repository for details:
+
management (DCIM) tool. See the source code repository for details:
 
https://github.com/netbox-community/netbox
 
https://github.com/netbox-community/netbox
You need to create an api token in netbox
+
You need to create an API token in NetBox to use it:
 
https://netbox.readthedocs.io/en/stable/api/authentication
 
https://netbox.readthedocs.io/en/stable/api/authentication
NetBox properties are:
+
The NetBox configuration properties are:
 
url
 
url
 
The REST API endpoint: http://yournetbox.domain.com/api
 
The REST API endpoint: http://yournetbox.domain.com/api
Line 263: Line 286:
 
DNS
 
DNS
 
The DNS plugin in Proxmox VE SDN is used to define a DNS API server for registration
 
The DNS plugin in Proxmox VE SDN is used to define a DNS API server for registration
of your hostname and IP-address. A DNS configuration is associated with one or
+
of your hostname and IP address. A DNS configuration is associated with one or
more zones, to provide DNS registration for all the sub-net IPs configured for
+
more zones, to provide DNS registration for all the subnet IPs configured for
 
a zone.
 
a zone.
PowerDNS plugin
+
PowerDNS Plugin
 
https://doc.powerdns.com/authoritative/http-api/index.html
 
https://doc.powerdns.com/authoritative/http-api/index.html
You need to enable the webserver and the API in your PowerDNS config:
+
You need to enable the web server and the API in your PowerDNS config:
 
api=yes
 
api=yes
 
api-key=arandomgeneratedstring
 
api-key=arandomgeneratedstring
 
webserver=yes
 
webserver=yes
 
webserver-port=8081
 
webserver-port=8081
Powerdns properties are:
+
The PowerDNS configuration options are:
 
url
 
url
 
The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
 
The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost
Line 282: Line 305:
 
Examples
 
Examples
 
VLAN Setup Example
 
VLAN Setup Example
While we show plain configuration content here, almost everything should
+
While we show plaintext configuration content here, almost everything
be configurable using the web-interface only.
+
should be configurable using the web-interface only.
 
Node1: /etc/network/interfaces
 
Node1: /etc/network/interfaces
 
auto vmbr0
 
auto vmbr0
Line 314: Line 337:
 
bridge: vmbr0
 
bridge: vmbr0
 
Create a VNet named &#8216;myvnet1' with `vlan-id` `10&#8217; and the previously created
 
Create a VNet named &#8216;myvnet1' with `vlan-id` `10&#8217; and the previously created
&#8216;myvlanzone&#8217; as it&#8217;s zone.
+
&#8216;myvlanzone&#8217; as its zone.
 
id: myvnet1
 
id: myvnet1
 
zone: myvlanzone
 
zone: myvlanzone
 
tag: 10
 
tag: 10
 
Apply the configuration through the main SDN panel, to create VNets locally on
 
Apply the configuration through the main SDN panel, to create VNets locally on
each nodes.
+
each node.
Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
+
Create a Debian-based virtual machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
 
auto eth0
 
auto eth0
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.100/24
 
         address 10.0.3.100/24
Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+
Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
 
&#8216;myvnet1&#8217; as vm1.
 
&#8216;myvnet1&#8217; as vm1.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 331: Line 354:
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.101/24
 
         address 10.0.3.101/24
Then, you should be able to ping between both VMs over that network.
+
Following this, you should be able to ping between both VMs over that network.
 
QinQ Setup Example
 
QinQ Setup Example
While we show plain configuration content here, almost everything should
+
While we show plaintext configuration content here, almost everything
be configurable using the web-interface only.
+
should be configurable using the web-interface only.
 
Node1: /etc/network/interfaces
 
Node1: /etc/network/interfaces
 
auto vmbr0
 
auto vmbr0
Line 361: Line 384:
 
         address 192.168.0.2/24
 
         address 192.168.0.2/24
 
source /etc/network/interfaces.d/*
 
source /etc/network/interfaces.d/*
Create an QinQ zone named &#8216;qinqzone1&#8217; with service VLAN 20
+
Create a QinQ zone named &#8216;qinqzone1&#8217; with service VLAN 20
 
id: qinqzone1
 
id: qinqzone1
 
bridge: vmbr0
 
bridge: vmbr0
Line 369: Line 392:
 
bridge: vmbr0
 
bridge: vmbr0
 
service vlan: 30
 
service vlan: 30
Create a VNet named &#8216;myvnet1&#8217; with customer vlan-id 100 on the previously
+
Create a VNet named &#8216;myvnet1&#8217; with customer VLAN-ID 100 on the previously
 
created &#8216;qinqzone1&#8217; zone.
 
created &#8216;qinqzone1&#8217; zone.
 
id: myvnet1
 
id: myvnet1
 
zone: qinqzone1
 
zone: qinqzone1
 
tag: 100
 
tag: 100
Create a &#8216;myvnet2&#8217; with customer VLAN-id 100 on the previously created
+
Create a &#8216;myvnet2&#8217; with customer VLAN-ID 100 on the previously created
 
&#8216;qinqzone2&#8217; zone.
 
&#8216;qinqzone2&#8217; zone.
 
id: myvnet2
 
id: myvnet2
Line 381: Line 404:
 
Apply the configuration on the main SDN web-interface panel to create VNets
 
Apply the configuration on the main SDN web-interface panel to create VNets
 
locally on each nodes.
 
locally on each nodes.
Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
+
Create a Debian-based virtual machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
 
auto eth0
 
auto eth0
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.100/24
 
         address 10.0.3.100/24
Create a second Virtual Machine (vm2) on node2, with a vNIC on the same VNet
+
Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet
 
&#8216;myvnet1&#8217; as vm1.
 
&#8216;myvnet1&#8217; as vm1.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 392: Line 415:
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.101/24
 
         address 10.0.3.101/24
Create a third Virtual Machine (vm3) on node1, with a vNIC on the other VNet
+
Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet
 
&#8216;myvnet2&#8217;.
 
&#8216;myvnet2&#8217;.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 398: Line 421:
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.102/24
 
         address 10.0.3.102/24
Create another Virtual Machine (vm4) on node2, with a vNIC on the same VNet
+
Create another virtual machine (vm4) on node2, with a vNIC on the same VNet
 
&#8216;myvnet2&#8217; as vm3.
 
&#8216;myvnet2&#8217; as vm3.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 404: Line 427:
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.103/24
 
         address 10.0.3.103/24
Then, you should be able to ping between the VMs vm1 and vm2, also
+
Then, you should be able to ping between the VMs vm1 and vm2, as well as
between vm3 and vm4. But, none of VMs vm1 or vm2 can ping the VMs vm3
+
between vm3 and vm4. However, neither of VMs vm1 or vm2 can ping VMs
or vm4, as they are on a different zone with different service-vlan.
+
vm3 or vm4, as they are on a different zone with a different service-vlan.
 
VXLAN Setup Example
 
VXLAN Setup Example
While we show plain configuration content here, almost everything should
+
While we show plaintext configuration content here, almost everything
be configurable using the web-interface only.
+
is configurable through the web-interface.
 
node1: /etc/network/interfaces
 
node1: /etc/network/interfaces
 
auto vmbr0
 
auto vmbr0
Line 440: Line 463:
 
         mtu 1500
 
         mtu 1500
 
source /etc/network/interfaces.d/*
 
source /etc/network/interfaces.d/*
Create an VXLAN zone named &#8216;myvxlanzone&#8217;, use the lower MTU to ensure the extra
+
Create a VXLAN zone named &#8216;myvxlanzone&#8217;, using a lower MTU to ensure the extra
 
50 bytes of the VXLAN header can fit. Add all previously configured IPs from
 
50 bytes of the VXLAN header can fit. Add all previously configured IPs from
the nodes as peer address list.
+
the nodes to the peer address list.
 
id: myvxlanzone
 
id: myvxlanzone
 
peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
 
peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
Line 453: Line 476:
 
Apply the configuration on the main SDN web-interface panel to create VNets
 
Apply the configuration on the main SDN web-interface panel to create VNets
 
locally on each nodes.
 
locally on each nodes.
Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
+
Create a Debian-based virtual machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
Use the following network configuration for this VM, note the lower MTU here.
+
Use the following network configuration for this VM (note the lower MTU).
 
auto eth0
 
auto eth0
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.3.100/24
 
         address 10.0.3.100/24
 
         mtu 1450
 
         mtu 1450
Create a second Virtual Machine (vm2) on node3, with a vNIC on the same VNet
+
Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet
 
&#8216;myvnet1&#8217; as vm1.
 
&#8216;myvnet1&#8217; as vm1.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 498: Line 521:
 
         mtu 1500
 
         mtu 1500
 
source /etc/network/interfaces.d/*
 
source /etc/network/interfaces.d/*
Create a EVPN controller, using a private ASN number and above node addreesses
+
Create an EVPN controller, using a private ASN number and the above node
as peers.
+
addresses as peers.
 
id: myevpnctl
 
id: myevpnctl
 
asn: 65000
 
asn: 65000
 
peers: 192.168.0.1,192.168.0.2,192.168.0.3
 
peers: 192.168.0.1,192.168.0.2,192.168.0.3
Create an EVPN zone named &#8216;myevpnzone&#8217; using the previously created
+
Create an EVPN zone named &#8216;myevpnzone&#8217;, using the previously created
EVPN-controller Define node1 and node2 as exit nodes.
+
EVPN-controller. Define node1 and node2 as exit nodes.
 
id: myevpnzone
 
id: myevpnzone
 
vrf vxlan tag: 10000
 
vrf vxlan tag: 10000
Line 515: Line 538:
 
zone: myevpnzone
 
zone: myevpnzone
 
tag: 11000
 
tag: 11000
Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on vnet1
+
Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on myvnet1.
 
subnet: 10.0.1.0/24
 
subnet: 10.0.1.0/24
 
gateway: 10.0.1.1
 
gateway: 10.0.1.1
Line 526: Line 549:
 
subnet: 10.0.2.0/24
 
subnet: 10.0.2.0/24
 
gateway: 10.0.2.1
 
gateway: 10.0.2.1
Apply the configuration on the main SDN web-interface panel to create VNets
+
Apply the configuration from the main SDN web-interface panel to create VNets
locally on each nodes and generate the FRR config.
+
locally on each node and generate the FRR config.
Create a Debian-based Virtual Machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
+
Create a Debian-based virtual machine (vm1) on node1, with a vNIC on &#8216;myvnet1&#8217;.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
 
auto eth0
 
auto eth0
Line 535: Line 558:
 
         gateway 10.0.1.1  #this is the ip of the vnet1
 
         gateway 10.0.1.1  #this is the ip of the vnet1
 
         mtu 1450
 
         mtu 1450
Create a second Virtual Machine (vm2) on node2, with a vNIC on the other VNet
+
Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet
 
&#8216;myvnet2&#8217;.
 
&#8216;myvnet2&#8217;.
 
Use the following network configuration for this VM:
 
Use the following network configuration for this VM:
Line 541: Line 564:
 
iface eth0 inet static
 
iface eth0 inet static
 
         address 10.0.2.100/24
 
         address 10.0.2.100/24
         gateway 10.0.2.1  #this is the ip of the vnet2
+
         gateway 10.0.2.1  #this is the ip of the myvnet2
 
         mtu 1450
 
         mtu 1450
 
Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
 
Then, you should be able to ping vm2 from vm1, and vm1 from vm2.
Line 548: Line 571:
 
nodes (node1 or node2) and from there it will leave those nodes over the
 
nodes (node1 or node2) and from there it will leave those nodes over the
 
default gateway configured on node1 or node2.
 
default gateway configured on node1 or node2.
Of course you need to add reverse routes for the 10.0.1.0/24 and
+
You need to add reverse routes for the 10.0.1.0/24 and 10.0.2.0/24
10.0.2.0/24 network to node1, node2 on your external gateway, so that the
+
networks to node1 and node2 on your external gateway, so that the public network
public network can reply back.
+
can reply back.
 
If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 
If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 
and 10.0.2.0/24 in this example), will be announced dynamically.
 
and 10.0.2.0/24 in this example), will be announced dynamically.
 
Notes
 
Notes
 
VXLAN IPSEC Encryption
 
VXLAN IPSEC Encryption
If you need to add encryption on top of VXLAN, it&#8217;s possible to do so with
+
If you need to add encryption on top of a VXLAN, it&#8217;s possible to do so with
IPSEC through strongswan. You&#8217;ll need to reduce the MTU by 60 bytes (IPv4)
+
IPSEC, through strongswan. You&#8217;ll need to reduce the MTU by 60 bytes (IPv4)
 
or 80 bytes (IPv6) to handle encryption.
 
or 80 bytes (IPv6) to handle encryption.
 
So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
 
So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC)
Line 562: Line 585:
 
Install strongswan
 
Install strongswan
 
apt install strongswan
 
apt install strongswan
Add configuration in &#8216;/etc/ipsec.conf&#8217;. We only need to encrypt traffic from
+
Add configuration to &#8216;/etc/ipsec.conf&#8217;. We only need to encrypt traffic from
 
the VXLAN UDP port 4789.
 
the VXLAN UDP port 4789.
 
conn %default
 
conn %default
Line 579: Line 602:
 
     authby=psk
 
     authby=psk
 
     auto=route
 
     auto=route
Then generate a preshared key with
+
Then generate a pre-shared key with:
 
openssl rand -base64 128
 
openssl rand -base64 128
and copy the key in &#8216;/etc/ipsec.secrets&#8217; so that the file content looks like:
+
and add the key to &#8216;/etc/ipsec.secrets&#8217;, so that the file contents looks like:
 
: PSK &lt;generatedbase64key&gt;
 
: PSK &lt;generatedbase64key&gt;
You need to copy the PSK and the config on other nodes.
+
You need to copy the PSK and the configuration onto the other nodes.
 
</pvehide>
 
</pvehide>
 
<!--PVE_IMPORT_END_MARKER-->
 
<!--PVE_IMPORT_END_MARKER-->

Latest revision as of 11:20, 4 May 2022

The Software Defined Network (SDN) feature allows you to create virtual networks (VNets) at the datacenter level.

Warning SDN is currently an experimental feature in Proxmox VE. This documentation for it is also still under development. Ask on our mailing lists or in the forum for questions and feedback.

Installation

To enable the experimental Software Defined Network (SDN) integration, you need to install the libpve-network-perl and ifupdown2 packages on every node:

apt update
apt install libpve-network-perl ifupdown2
Note Proxmox VE version 7 and above come installed with ifupdown2.

After this, you need to add the following line to the end of the /etc/network/interfaces configuration file, so that the SDN configuration gets included and activated.

source /etc/network/interfaces.d/*

Basic Overview

The Proxmox VE SDN allows for separation and fine-grained control of virtual guest networks, using flexible, software-controlled configurations.

Separation is managed through zones, where a zone is its own virtual separated network area. A VNet is a type of a virtual network connected to a zone. Depending on which type or plugin the zone uses, it can behave differently and offer different features, advantages, and disadvantages. Normally, a VNet appears as a common Linux bridge with either a VLAN or VXLAN tag, however, some can also use layer 3 routing for control. VNets are deployed locally on each node, after being configured from the cluster-wide datacenter SDN administration interface.

Main Configuration

Configuration is done at the datacenter (cluster-wide) level and is saved in files located in the shared configuration file system: /etc/pve/sdn

On the web-interface, SDN features 3 main sections:

  • SDN: An overview of the SDN state

  • Zones: Create and manage the virtually separated network zones

  • VNets: Create virtual network bridges and manage subnets

In addition to this, the following options are offered:

  • Controller: For controlling layer 3 routing in complex setups

  • Subnets: Used to defined IP networks on VNets

  • IPAM: Enables the use of external tools for IP address management (guest IPs)

  • DNS: Define a DNS server API for registering virtual guests' hostname and IP addresses

SDN

This is the main status panel. Here you can see the deployment status of zones on different nodes.

The Apply button is used to push and reload local configuration on all cluster nodes.

Local Deployment Monitoring

After applying the configuration through the main SDN panel, the local network configuration is generated locally on each node in the file /etc/network/interfaces.d/sdn, and reloaded with ifupdown2.

You can monitor the status of local zones and VNets through the main tree.

Zones

A zone defines a virtually separated network. Zones can be restricted to specific nodes and assigned permissions, in order to restrict users to a certain zone and its contained VNets.

Different technologies can be used for separation:

  • VLAN: Virtual LANs are the classic method of subdividing a LAN

  • QinQ: Stacked VLAN (formally known as IEEE 802.1ad)

  • VXLAN: Layer2 VXLAN

  • Simple: Isolated Bridge. A simple layer 3 routing bridge (NAT)

  • EVPN (BGP EVPN): VXLAN using layer 3 border gateway protocol (BGP) routing

Common options

The following options are available for all zone types:

nodes

The nodes which the zone and associated VNets should be deployed on

ipam

Optional. Use an IP Address Management (IPAM) tool to manage IPs in the zone.

dns

Optional. DNS API server.

reversedns

Optional. Reverse DNS API server.

dnszone

Optional. DNS domain name. Used to register hostnames, such as <hostname>.<domain>. The DNS zone must already exist on the DNS server.

Simple Zones

This is the simplest plugin. It will create an isolated VNet bridge. This bridge is not linked to a physical interface, and VM traffic is only local between the node(s). It can also be used in NAT or routed setups.

VLAN Zones

This plugin reuses an existing local Linux or OVS bridge, and manages the VLANs on it. The benefit of using the SDN module is that you can create different zones with specific VNet VLAN tags, and restrict virtual machines to separated zones.

Specific VLAN configuration options:

bridge

Reuse this local bridge or OVS switch, already configured on each local node.

QinQ Zones

QinQ also known as VLAN stacking, wherein the first VLAN tag is defined for the zone (the service-vlan), and the second VLAN tag is defined for the VNets.

Note Your physical network switches must support stacked VLANs for this configuration!

Below are the configuration options specific to QinQ:

bridge

A local, VLAN-aware bridge that is already configured on each local node

service vlan

The main VLAN tag of this zone

service vlan protocol

Allows you to choose between an 802.1q (default) or 802.1ad service VLAN type.

mtu

Due to the double stacking of tags, you need 4 more bytes for QinQ VLANs. For example, you must reduce the MTU to 1496 if you physical interface MTU is 1500.

VXLAN Zones

The VXLAN plugin establishes a tunnel (overlay) on top of an existing network (underlay). This encapsulates layer 2 Ethernet frames within layer 4 UDP datagrams, using 4789 as the default destination port. You can, for example, create a private IPv4 VXLAN network on top of public internet network nodes.

This is a layer 2 tunnel only, so no routing between different VNets is possible.

Each VNet will have a specific VXLAN ID in the range 1 - 16777215.

Specific EVPN configuration options:

peers address list

A list of IP addresses from each node through which you want to communicate. Can also be external nodes.

mtu

Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes lower than the outgoing physical interface.

EVPN Zones

This is the most complex of all the supported plugins.

BGP-EVPN allows you to create a routable layer 3 network. The VNet of EVPN can have an anycast IP address and/or MAC address. The bridge IP is the same on each node, meaning a virtual guest can use this address as gateway.

Routing can work across VNets from different zones through a VRF (Virtual Routing and Forwarding) interface.

The configuration options specific to EVPN are as follows:

VRF VXLAN tag

This is a VXLAN-ID used for routing interconnect between VNets. It must be different than the VXLAN-ID of the VNets.

controller

An EVPN-controller must to be defined first (see controller plugins section).

VNet MAC address

A unique, anycast MAC address for all VNets in this zone. Will be auto-generated if not defined.

Exit Nodes

Optional. This is used if you want to define some Proxmox VE nodes as exit gateways from the EVPN network, through the real network. The configured nodes will announce a default route in the EVPN network.

Primary Exit Node

Optional. If you use multiple exit nodes, this forces traffic to a primary exit node, instead of load-balancing on all nodes. This is required if you want to use SNAT or if your upstream router doesn’t support ECMP.

Exit Nodes local routing

Optional. This is a special option if you need to reach a VM/CT service from an exit node. (By default, the exit nodes only allow forwarding traffic between real network and EVPN network).

Advertise Subnets

Optional. If you have silent VMs/CTs (for example, if you have multiple IPs and the anycast gateway doesn’t see traffic from theses IPs, the IP addresses won’t be able to be reach inside the EVPN network). This option will announce the full subnet in the EVPN network in this case.

Disable Arp-Nd Suppression

Optional. Don’t suppress ARP or ND packets. This is required if you use floating IPs in your guest VMs (IP are MAC addresses are being moved between systems).

Route-target import

Optional. Allows you to import a list of external EVPN route targets. Used for cross-DC or different EVPN network interconnects.

MTU

Because VXLAN encapsulation uses 50 bytes, the MTU needs to be 50 bytes less than the maximal MTU of the outgoing physical interface.

VNets

A VNet is, in its basic form, a Linux bridge that will be deployed locally on the node and used for virtual machine communication.

The VNet configuration properties are:

ID

An 8 character ID to name and identify a VNet

Alias

Optional longer name, if the ID isn’t enough

Zone

The associated zone for this VNet

Tag

The unique VLAN or VXLAN ID

VLAN Aware

Enable adding an extra VLAN tag in the virtual machine or container’s vNIC configuration, to allow the guest OS to manage the VLAN’s tag.

Subnets

A subnetwork (subnet) allows you to define a specific IP network (IPv4 or IPv6). For each VNet, you can define one or more subnets.

A subnet can be used to:

  • Restrict the IP addresses you can define on a specific VNet

  • Assign routes/gateways on a VNet in layer 3 zones

  • Enable SNAT on a VNet in layer 3 zones

  • Auto assign IPs on virtual guests (VM or CT) through IPAM plugins

  • DNS registration through DNS plugins

If an IPAM server is associated with the subnet zone, the subnet prefix will be automatically registered in the IPAM.

Subnet properties are:

ID

A CIDR network address, for example 10.0.0.0/8

Gateway

The IP address of the network’s default gateway. On layer 3 zones (Simple/EVPN plugins), it will be deployed on the VNet.

SNAT

Optional. Enable SNAT for layer 3 zones (Simple/EVPN plugins), for this subnet. The subnet’s source IP will be NATted to server’s outgoing interface/IP. On EVPN zones, this is only done on EVPN gateway-nodes.

Dnszoneprefix

Optional. Add a prefix to the domain registration, like <hostname>.prefix.<domain>

Controllers

Some zone types need an external controller to manage the VNet control-plane. Currently this is only required for the bgp-evpn zone plugin.

EVPN Controller

For BGP-EVPN, we need a controller to manage the control plane. The currently supported software controller is the "frr" router. You may need to install it on each node where you want to deploy EVPN zones.

apt install frr frr-pythontools

Configuration options:

asn

A unique BGP ASN number. It’s highly recommended to use a private ASN number (64512 – 65534, 4200000000 – 4294967294), as otherwise you could end up breaking global routing by mistake.

peers

An IP list of all nodes where you want to communicate for the EVPN (could also be external nodes or route reflectors servers)

BGP Controller

The BGP controller is not used directly by a zone. You can use it to configure FRR to manage BGP peers.

For BGP-EVPN, it can be used to define a different ASN by node, so doing EBGP.

Configuration options:

node

The node of this BGP controller

asn

A unique BGP ASN number. It’s highly recommended to use a private ASN number in the range (64512 - 65534) or (4200000000 - 4294967294), as otherwise you could break global routing by mistake.

peers

A list of peer IP addresses you want to communicate with using the underlying BGP network.

ebgp

If your peer’s remote-AS is different, this enables EBGP.

loopback

Use a loopback or dummy interface as the source of the EVPN network (for multipath).

ebgp-mutltihop

Increase the number of hops to reach peers, in case they are not directly connected or they use loopback.

bgp-multipath-as-path-relax

Allow ECMP if your peers have different ASN.

IPAMs

IPAM (IP Address Management) tools are used to manage/assign the IP addresses of guests on the network. It can be used to find free IP addresses when you create a VM/CT for example (not yet implemented).

An IPAM can be associated with one or more zones, to provide IP addresses for all subnets defined in those zones.

Proxmox VE IPAM Plugin

This is the default internal IPAM for your Proxmox VE cluster, if you don’t have external IPAM software.

phpIPAM Plugin

You need to create an application in phpIPAM and add an API token with admin privileges.

The phpIPAM configuration properties are:

url

The REST-API endpoint: http://phpipam.domain.com/api/<appname>/

token

An API access token

section

An integer ID. Sections are a group of subnets in phpIPAM. Default installations use sectionid=1 for customers.

NetBox IPAM Plugin

NetBox is an IP address management (IPAM) and datacenter infrastructure management (DCIM) tool. See the source code repository for details: https://github.com/netbox-community/netbox

You need to create an API token in NetBox to use it: https://netbox.readthedocs.io/en/stable/api/authentication

The NetBox configuration properties are:

url

The REST API endpoint: http://yournetbox.domain.com/api

token

An API access token

DNS

The DNS plugin in Proxmox VE SDN is used to define a DNS API server for registration of your hostname and IP address. A DNS configuration is associated with one or more zones, to provide DNS registration for all the subnet IPs configured for a zone.

PowerDNS Plugin

You need to enable the web server and the API in your PowerDNS config:

api=yes
api-key=arandomgeneratedstring
webserver=yes
webserver-port=8081

The PowerDNS configuration options are:

url

The REST API endpoint: http://yourpowerdnserver.domain.com:8081/api/v1/servers/localhost

key

An API access key

ttl

The default TTL for records

Examples

VLAN Setup Example

Tip While we show plaintext configuration content here, almost everything should be configurable using the web-interface only.

Node1: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.0.1/24

source /etc/network/interfaces.d/*

Node2: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.0.2/24

source /etc/network/interfaces.d/*

Create a VLAN zone named ‘myvlanzone’:

id: myvlanzone
bridge: vmbr0

Create a VNet named ‘myvnet1' with `vlan-id` `10’ and the previously created ‘myvlanzone’ as its zone.

id: myvnet1
zone: myvlanzone
tag: 10

Apply the configuration through the main SDN panel, to create VNets locally on each node.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on ‘myvnet1’.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.100/24

Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet ‘myvnet1’ as vm1.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.101/24

Following this, you should be able to ping between both VMs over that network.

QinQ Setup Example

Tip While we show plaintext configuration content here, almost everything should be configurable using the web-interface only.

Node1: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.0.1/24

source /etc/network/interfaces.d/*

Node2: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet manual
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

#management ip on vlan100
auto vmbr0.100
iface vmbr0.100 inet static
        address 192.168.0.2/24

source /etc/network/interfaces.d/*

Create a QinQ zone named ‘qinqzone1’ with service VLAN 20

id: qinqzone1
bridge: vmbr0
service vlan: 20

Create another QinQ zone named ‘qinqzone2’ with service VLAN 30

id: qinqzone2
bridge: vmbr0
service vlan: 30

Create a VNet named ‘myvnet1’ with customer VLAN-ID 100 on the previously created ‘qinqzone1’ zone.

id: myvnet1
zone: qinqzone1
tag: 100

Create a ‘myvnet2’ with customer VLAN-ID 100 on the previously created ‘qinqzone2’ zone.

id: myvnet2
zone: qinqzone2
tag: 100

Apply the configuration on the main SDN web-interface panel to create VNets locally on each nodes.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on ‘myvnet1’.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.100/24

Create a second virtual machine (vm2) on node2, with a vNIC on the same VNet ‘myvnet1’ as vm1.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.101/24

Create a third virtual machine (vm3) on node1, with a vNIC on the other VNet ‘myvnet2’.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.102/24

Create another virtual machine (vm4) on node2, with a vNIC on the same VNet ‘myvnet2’ as vm3.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.103/24

Then, you should be able to ping between the VMs vm1 and vm2, as well as between vm3 and vm4. However, neither of VMs vm1 or vm2 can ping VMs vm3 or vm4, as they are on a different zone with a different service-vlan.

VXLAN Setup Example

Tip While we show plaintext configuration content here, almost everything is configurable through the web-interface.

node1: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.1/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

node2: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.2/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

node3: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.3/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

Create a VXLAN zone named ‘myvxlanzone’, using a lower MTU to ensure the extra 50 bytes of the VXLAN header can fit. Add all previously configured IPs from the nodes to the peer address list.

id: myvxlanzone
peers address list: 192.168.0.1,192.168.0.2,192.168.0.3
mtu: 1450

Create a VNet named ‘myvnet1’ using the VXLAN zone ‘myvxlanzone’ created previously.

id: myvnet1
zone: myvxlanzone
tag: 100000

Apply the configuration on the main SDN web-interface panel to create VNets locally on each nodes.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on ‘myvnet1’.

Use the following network configuration for this VM (note the lower MTU).

auto eth0
iface eth0 inet static
        address 10.0.3.100/24
        mtu 1450

Create a second virtual machine (vm2) on node3, with a vNIC on the same VNet ‘myvnet1’ as vm1.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.3.101/24
        mtu 1450

Then, you should be able to ping between between vm1 and vm2.

EVPN Setup Example

node1: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.1/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

node2: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.2/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

node3: /etc/network/interfaces

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.3/24
        gateway 192.168.0.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        mtu 1500

source /etc/network/interfaces.d/*

Create an EVPN controller, using a private ASN number and the above node addresses as peers.

id: myevpnctl
asn: 65000
peers: 192.168.0.1,192.168.0.2,192.168.0.3

Create an EVPN zone named ‘myevpnzone’, using the previously created EVPN-controller. Define node1 and node2 as exit nodes.

id: myevpnzone
vrf vxlan tag: 10000
controller: myevpnctl
mtu: 1450
vnet mac address: 32:F4:05:FE:6C:0A
exitnodes: node1,node2

Create the first VNet named ‘myvnet1’ using the EVPN zone ‘myevpnzone’.

id: myvnet1
zone: myevpnzone
tag: 11000

Create a subnet 10.0.1.0/24 with 10.0.1.1 as gateway on myvnet1.

subnet: 10.0.1.0/24
gateway: 10.0.1.1

Create the second VNet named ‘myvnet2’ using the same EVPN zone ‘myevpnzone’, a different IPv4 CIDR network.

id: myvnet2
zone: myevpnzone
tag: 12000

Create a different subnet 10.0.2.0/24 with 10.0.2.1 as gateway on vnet2

subnet: 10.0.2.0/24
gateway: 10.0.2.1

Apply the configuration from the main SDN web-interface panel to create VNets locally on each node and generate the FRR config.

Create a Debian-based virtual machine (vm1) on node1, with a vNIC on ‘myvnet1’.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.1.100/24
        gateway 10.0.1.1   #this is the ip of the vnet1
        mtu 1450

Create a second virtual machine (vm2) on node2, with a vNIC on the other VNet ‘myvnet2’.

Use the following network configuration for this VM:

auto eth0
iface eth0 inet static
        address 10.0.2.100/24
        gateway 10.0.2.1   #this is the ip of the myvnet2
        mtu 1450

Then, you should be able to ping vm2 from vm1, and vm1 from vm2.

If you ping an external IP from vm2 on the non-gateway node3, the packet will go to the configured myvnet2 gateway, then will be routed to the exit nodes (node1 or node2) and from there it will leave those nodes over the default gateway configured on node1 or node2.

Note You need to add reverse routes for the 10.0.1.0/24 and 10.0.2.0/24 networks to node1 and node2 on your external gateway, so that the public network can reply back.

If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24 and 10.0.2.0/24 in this example), will be announced dynamically.

Notes

VXLAN IPSEC Encryption

If you need to add encryption on top of a VXLAN, it’s possible to do so with IPSEC, through strongswan. You’ll need to reduce the MTU by 60 bytes (IPv4) or 80 bytes (IPv6) to handle encryption.

So with default real 1500 MTU, you need to use a MTU of 1370 (1370 + 80 (IPSEC) + 50 (VXLAN) == 1500).

Install strongswan
apt install strongswan

Add configuration to ‘/etc/ipsec.conf’. We only need to encrypt traffic from the VXLAN UDP port 4789.

conn %default
    ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules

conn output
    rightsubnet=%dynamic[udp/4789]
    right=%any
    type=transport
    authby=psk
    auto=route

conn input
    leftsubnet=%dynamic[udp/4789]
    type=transport
    authby=psk
    auto=route

Then generate a pre-shared key with:

openssl rand -base64 128

and add the key to ‘/etc/ipsec.secrets’, so that the file contents looks like:

: PSK <generatedbase64key>

You need to copy the PSK and the configuration onto the other nodes.