Open vSwitch: Difference between revisions
Brad House (talk | contribs) |
Brad House (talk | contribs) |
||
Line 32: | Line 32: | ||
== Configuration == | == Configuration == | ||
Official reference here, though a bit bare: http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob;f=debian/openvswitch-switch.README.Debian;hb=HEAD | |||
=== Overview === | === Overview === |
Revision as of 11:36, 14 October 2014
Installation
- Install the Open vSwitch packages
apt-get install openvswitch-switch
Startup Workaround
- There appears to be a bug in the current openvswitch package. It expects /run/network/ifstate to exist, but that file is created by the network scripts and openvswitch starts before network. So we need to ensure that file exists prior to starting openvswitch. That can be done by cut and pasting the below into your terminal which will append this check to /etc/default/openvswitch-switch:
cat >> /etc/default/openvswitch-switch << 'EOF' RUN_DIR="/run/network" IFSTATE="$RUN_DIR/ifstate" check_ifstate() { if [ ! -d "$RUN_DIR" ] ; then if ! mkdir -p "$RUN_DIR" ; then log_failure_msg "can't create $RUN_DIR" exit 1 fi fi if [ ! -r "$IFSTATE" ] ; then if ! :> "$IFSTATE" ; then log_failure_msg "can't initialise $IFSTATE" exit 1 fi fi } check_ifstate EOF
Configuration
Official reference here, though a bit bare: http://git.openvswitch.org/cgi-bin/gitweb.cgi?p=openvswitch;a=blob;f=debian/openvswitch-switch.README.Debian;hb=HEAD
Overview
Open vSwitch and Linux bonding and bridging or vlans MUST NOT be mixed. For instance, do not attempt to add a vlan to an OVS Bond, or add a Linux Bond to an OVSBridge or vice-versa. Open vSwitch is specifically tailored to function within virtualized environments, there is no reason to use the native linux functionality.
Bridges
A bridge is another term for a Switch. It directs traffic to the appropriate interface based on mac address. Open vSwitch bridges should contain raw ethernet devices, along with virtual interfaces such as OVSBonds or OVSIntPorts. These bridges can carry multiple vlans, and be broken out into 'internal ports' to be used as vlan interfaces on the host.
It should be noted that it is recommended that the bridge is bound to a trunk port with no untagged vlans. This means that your bridge itself will never have an ip address. You would then split out your VLANs using virtual interfaces (OVSIntPort). Proxmox will also assign the guest VMs a tap interface associated with the assigned vlan, so you do not need a bridge per vlan. You should treat your OVSBridge much like a physical hardware switch.
When configuring a bridge, in /etc/network/interfaces, prefix the bridge interface definition with allow-ovs $iface. For instance, a simple bridge containing a single interface would look like:
auto vmbr0 allow-ovs vmbr0 iface vmbr0 inet manual ovs_type OVSBridge ovs_ports eth0
However, if you must have _untagged_ traffic on that interface (beyond your control), and you need to use it from the local host (and thus assign an ip address), you would assign it directly to the bridge:
auto vmbr0 allow-ovs vmbr0 iface vmbr0 inet static ovs_type OVSBridge ovs_ports eth0 address 10.1.1.5 netmask 255.255.255.0 gateway 10.1.1.1
Remember, if you want to split out other vlans than the native for use on the local host, you should use OVSIntPorts, see sections to follow.
You do not need to explicitly have definitions for physical interfaces such as eth0 in the configuration, they will be automatically brought up. However, any virtual interfaces (OVSBonds or OVSIntPorts) should have their definitions prefixed with allow-$brname $iface, e.g. allow-vmbr0 bond0
NOTE: All interfaces must be listed under ovs_ports that are part of the bridge even if you have a port definition (e.g. OVSIntPort) that cross-references the bridge!!!
Bonds
Bonds are used to join multiple network interfaces together to act as single unit. Bonds must refer to raw ethernet devices (e.g. eth0, eth1).
When configuring a bond, it is recommended to use LACP (aka 802.3ad) for link aggregation. This requires switch support on the other end. A simple bond using eth0 and eth1 that will be part of the vmbr0 bridge might look like this.
allow-vmbr0 bond0 iface ovsbond inet manual ovs_bridge vmbr0 ovs_type OVSBond ovs_bonds eth0 eth1 ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast
VLANs Host Interfaces
In order for the host (e.g. proxmox host, not VMs themselves!) to utilize a vlan within the bridge, you must create OVSIntPorts. These split out an untagged virtual interface in the specified vlan that you can assign an ip address to (or use DHCP). You need to set ovs_options tag=$VLAN to let OVS know what vlan the interface should be a part of. In the switch world, this is commonly referred to as an RVI (Routed Virtual Interface).
IMPORTANT: These OVSIntPorts you create MUST also show up in the actual bridge definition under ovs_ports. If they do not, they will NOT be brought up even though you specified an ovs_bridge. You also need to prefix the definition with allow-$bridge $iface
Setting up this vlan port would look like this in /etc/network/interfaces:
allow-vmbr0 vlan50 iface vlan50 inet static ovs_type OVSIntPort ovs_bridge vmbr0 ovs_options tag=50 ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif address 10.50.10.44 netmask 255.255.255.0 gateway 10.50.10.1
Note on MTU
If you plan on using a MTU larger than the default of 1500, you need to mark any physical interfaces, bonds, and bridges with a larger MTU by adding an mtu setting to the definition such as mtu 9000 otherwise it will be disallowed. However, you should NOT create definitions for your physical interfaces, instead, at the bridge or bond layer, you should use a pre-up script such as
pre-up ( ifconfig eth0 mtu 9000 && ifconfig eth1 mtu 9000 )
If you instead create entries in /etc/network/interfaces for those physical interfaces and set the MTU there, then that MTU will propagate to EVERY child. That means you wouldn't be able to configure OVSIntPorts with an mtu of 1500.
Example
The below example shows you a combination of all the above features. 2 NICs are bonded together and added to an OVS Bridge. 2 vlan interfaces are split out in order to provide the host access to vlans with different MTUs.
This is a complete and working /etc/network/interfaces listing:
# Loopback interface auto lo iface lo inet loopback # Bond eth0 and eth1 together allow-vmbr0 bond0 iface bond0 inet manual ovs_bridge vmbr0 ovs_type OVSBond ovs_bonds eth0 eth1 # Force the MTU of the physical interfaces to be jumbo-frame capable. # This doesn't mean that any OVSIntPorts must be jumbo-capable. # We cannot, however set up definitions for eth0 and eth1 directly due # to what appear to be bugs in the initialization process. pre-up ( ifconfig eth0 mtu 9000 && ifconfig eth1 mtu 9000 ) ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast mtu 9000 # Bridge for our bond and vlan virtual interfaces (our VMs will # also attach to this bridge) auto vmbr0 allow-ovs vmbr0 iface vmbr0 inet manual ovs_type OVSBridge # NOTE: we MUST mention bond0, vlan50, and vlan55 even though each # of them lists ovs_bridge vmbr0! Not sure why it needs this # kind of cross-referencing but it won't work without it! ovs_ports bond0 vlan50 vlan55 mtu 9000 # Proxmox cluster communication vlan allow-vmbr0 vlan50 iface vlan50 inet static ovs_type OVSIntPort ovs_bridge vmbr0 ovs_options tag=50 ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif address 10.50.10.44 netmask 255.255.255.0 gateway 10.50.10.1 mtu 1500 # Ceph cluster communication vlan (jumbo frames) allow-vmbr0 vlan55 iface vlan55 inet static ovs_type OVSIntPort ovs_bridge vmbr0 ovs_options tag=55 ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif address 10.55.10.44 netmask 255.255.255.0 mtu 9000
Multicast
Right now Open vSwitch doesn't do anything in regards to multicast. Typically where you might tell linux to enable the multicast querier on the bridge, you should instead set up your querier at your router or switch. Please refer to the Multicast_notes wiki for more information.
Using Open vSwitch in Proxmox
Using Open vSwitch isn't that much different than using normal linux bridges. The main difference is instead of having a bridge per vlan, you have a single bridge containing all your vlans. Then when configuring the network interface for the VM, you would select the bridge (probably the only bridge you have), and you would also enter the VLAN Tag associated with the VLAN you want your VM to be a part of. Now there is zero effort when adding or removing VLANs!