[PVE-User] network conversion mesh network into lacp bonding setup

[Ronny Aasen]

On 26.09.2018 17:26, lists wrote:
> Hi all,
>
> I am going to make some change to our proxmox networking, and I'd like 
> some fresh eyes to take a look at my plans... :-)
>
> We have three pve hosts, ceph network is a meshed 10G network setup, 
> directly connecting the pve hosts to each other. Client access is on a 
> 'regular' ip 1G NIC.
>
> Sample /etc/network/interfaces (server pve10) for current config:
>
> # client access
> auto vmbr0
> iface vmbr0 inet static
>         address  192.168.89.10
>         netmask  255.255.255.0
>         gateway  192.168.89.1
>         bridge_ports eth0
>         bridge_stp off
>         bridge_fd 0
>
> # to pve2/ceph
> auto eth2
> iface eth2 inet static
>         address  10.10.89.1
>         netmask  255.255.255.0
>         mtu 9000
>         up route add -net 10.10.89.2 netmask 255.255.255.255 dev eth2
>         down route del -net 10.10.89.2 netmask 255.255.255.255 dev eth2
>
> # to pve3/ceph
> auto eth3
> iface eth3 inet static
>         address  10.10.89.1
>         netmask  255.255.255.0
>         mtu 9000
>         up route add -net 10.10.89.3 netmask 255.255.255.255 dev eth3
>         down route del -net 10.10.89.3 netmask 255.255.255.255 dev eth3
>
> See for more info on the meshed network: 
> https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server. It 
> works very nicely.
>
> Now, we want to change networking from the above to: dual 10G lacp 
> bonds per server to our hp procurve chassis.
>
> So, in order to change as little as possible, I would like to keep 
> ceph config the same, meaning: retain all IPs/config, and use 
> something like this:
>
> auto bond0
> iface bond0 inet manual
>       slaves eth2 eth3
>       bond_miimon 100
>       bond_mode 802.3ad
>       bond_xmit_hash_policy layer3+4
>
> allow-hotplug vmbr0
> auto vmbr0
>
> # client access
> iface vmbr0 inet static
>         address  192.168.89.10
>         netmask  255.255.255.0
>         gateway  192.168.89.1
>         bridge_ports bond0
>         bridge_stp off
>         bridge_fd 0
>
> # ip for access to other cephs
> iface vmbr0 inet static
>     address 10.10.89.1
>     netmask 255.255.255.0
>
> Then cable eth2 / eth3 to the lacp ports on the HP procurve.
>
> I am assuming that this would make all traffic (ceph and VMs) float 
> over the same two 10G lacp wires, and both ceph and VMs would not 
> notice any difference. I'm also assuming that no other config changes 
> would be required at all.
>
> So, any errors in the above reasoning? I realise we cannot have jumbo 
> frames in this setup, but I don't think I mind. I also realise that 
> currently we have seperated ceph and VMs traffic, and in the new 
> situation we don't anymore, but this seems accepted (perhaps even 
> recommended) for small networks like ours at the ceph mailinglist 
> nowadays.
>
> So... feedback to all of the above please... :-) 



multiple identical stancas like
iface vmbr0 inet static
will likely fail.
you can (but should not)  add multiple ip addresses on a interface but 
use something like
up ip addr add 1.2.3.4/24 dev vmbr0  on the first entry.

what i do is...
bundle physical links together with lacp bond like you do here.
run multiple vlans over the physical bond. (i have vm's in many 
different vlans)
have a vlan aware bridge.
use vlans for ip addresses for the vlan aware bridge.
use the vlan tag in the vm config to connect a vm to a given vlan on the 
vlan aware bridge

but if you do not need the added complexity of the vlan aware bridge you 
can do something like...


iface eth0 inet manual
         mtu 9000

iface eth1 inet manual
         mtu 9000

#bond of interfaces
auto bond0
iface bond0 inet manual
         slaves eth0 eth1
         bond miimon 100
         bond_mode 802.3ad
         bond_xmit_hash_policy layer2+3
         mtu 9000

#vmbr bridge for vm's on vlan 10 (bond0.10)
auto vmbr0
iface vmbr0 inet static
         address  192.168.89.10
         netmask  255.255.255.0
         gateway  192.168.89.1
         bridge_ports bond0.10
         bridge_stp off
         bridge_fd 0

ip interface for ceph
auto bond0.20
iface bond0.20  inet static
     address 10.10.89.1
     netmask 255.255.255.0




good luck
Ronny Aasen