[PVE-User] Multicast problems with Intel X540 - 10Gtek network card?
Eneko Lacunza
elacunza at binovo.es
Wed Dec 5 13:55:05 CET 2018
Hi Stefan,
Thanks a lot for your suggestions!
Cheers
Eneko
El 5/12/18 a las 13:52, Stefan M. Radman escribió:
> Hi Eneko
>
> Even if PVE does it by default it does not mean that it is the best thing to do.
> The default configuration of PVE is a starting point.
>
> https://pve.proxmox.com/wiki/Separate_Cluster_Network
> Separate Cluster Network - Introduction<https://pve.proxmox.com/wiki/Separate_Cluster_Network#Introduction>
> It is good practice to use a separate network for corosync, which handles the cluster communication in Proxmox VE.
> It is one of the most important part in an fault tolerant (HA) system and other network traffic may disturb corosync.
>
> I'd recommend a thorough reading of the document quoted above.
>
> Don't use vmbr0 for cluster traffic.
> Don't use any vmbr for cluster traffic.
>
> Stefan
>
> On Dec 5, 2018, at 13:34, Eneko Lacunza <elacunza at binovo.es<mailto:elacunza at binovo.es>> wrote:
>
> Hi Stefan,
>
> El 4/12/18 a las 23:50, Stefan M. Radman escribió:
> Don't put your corosync traffic on bridges.
> Dedicate an untagged interface on each node for corosync.
> This is surprising, because Proxmox does this by default! :)
>
> All you need for your cluster network is this:
>
> auto eth3
> iface eth3 inet static
> address 192.168.10.201
> netmask 255.255.255.0
> #corosync ring0
>
> Put that interface into an isolated VLAN with IGMP snooping enabled.
> Prune that VLAN from all trunks to limit its extent and your troubles.
>
> But we're no using eth3/vmbr10 for corosync. We're using vmbr0 (default Proxmox config) por corosync.
>
> Thanks a lot
> Eneko
>
> Stefan
>
>
> On Dec 4, 2018, at 8:03 PM, Ronny Aasen <ronny+pve-user at aasen.cx<mailto:ronny+pve-user at aasen.cx><mailto:ronny+pve-user at aasen.cx>> wrote:
>
> vmbr10 is a bridge (or as switch by another name)
> if you want the switch to work reliably with multicast you probably need to enable multicast querier.
> |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier|
>
> or you can disable snooping, so that it treats multicast as broadcast. |
> echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping|
>
> this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage.
> https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier
>
> kind regards
> Ronny Aasen
>
>
>
> On 04.12.2018 17:54, Eneko Lacunza wrote:
> Hi all,
>
> Seems I found the solution.
>
> eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end.
>
> I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :)
>
> It is strange because multicast is on VLAN 1 network...
>
> Cheers and thanks a lot
> Eneko
>
> El 4/12/18 a las 16:18, Eneko Lacunza escribió:
>
> hi Marcus,
>
> El 4/12/18 a las 16:09, Marcus Haarmann escribió:
> Hi,
>
> you did not provide details about your configuration.
> How is the network card set up ? Bonding ?
> Send your /etc/network/interfaces details.
> If bonding is active, check if the mode is correct in /proc/net/bonding.
> We encountered differences between /etc/network/interfaces setup and resulting mode.
> Also, check your switch configuration, VLAN setup, MTU etc.
> Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration.
>
> /etc/network/interfaces in proxmox1 node:
> auto lo
> iface lo inet loopback
> iface eth0 inet manual
> iface eth1 inet manual
> iface eth2 inet manual
> iface eth3 inet manual
> iface eth4 inet manual
> iface eth5 inet manual
>
> auto vmbr10
> iface vmbr10 inet static
> address 192.168.10.201
> netmask 255.255.255.0
> bridge_ports eth3
> bridge_stp off
> bridge_fd 0
>
> auto vmbr0
> iface vmbr0 inet static
> address 192.168.0.201
> netmask 255.255.255.0
> gateway 192.168.0.100
> bridge_ports eth4
> bridge_stp off
> bridge_fd 0
>
> auto eth4.100
> iface eth4.100 inet static
> address 10.0.2.1
> netmask 255.255.255.0
> up ip addr add 10.0.3.1/24 dev eth4.100
>
> Cluster is running on vmbr0 network (192.168.0.0/24)
>
> Cheers
>
>
> Marcus Haarmann
>
>
> Von: "Eneko Lacunza" <elacunza at binovo.es<mailto:elacunza at binovo.es><mailto:elacunza at binovo.es>>
> An: "pve-user" <pve-user at pve.proxmox.com<mailto:pve-user at pve.proxmox.com><mailto:pve-user at pve.proxmox.com>>
> Gesendet: Dienstag, 4. Dezember 2018 15:57:10
> Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card?
>
> Hi all,
>
> We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph
> hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the
> three Proxmox nodes is new too :)
>
> Generally all was good and VMs are working well. :-)
>
> BUT, we have some problems with the cluster; promxox1 node joins and
> then after about 4 minutes drops from the cluster.
>
> All multicast tests
> https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast
> run fine except the last one:
>
> *** proxmox1:
>
> root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4
>
> proxmox3 : waiting for response msg
>
> proxmox4 : waiting for response msg
>
> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox3 : given amount of query messages was sent
>
> proxmox4 : given amount of query messages was sent
>
> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061
>
> proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068
>
> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059
>
> proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064
>
>
> *** proxmox3:
>
> root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4
>
> proxmox1 : waiting for response msg
>
> proxmox4 : waiting for response msg
>
> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox1 : waiting for response msg
>
> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox4 : given amount of query messages was sent
>
> proxmox1 : given amount of query messages was sent
>
> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055
>
> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054
>
> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026
>
> proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030
>
>
> *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4
>
> proxmox1 : waiting for response msg
>
> proxmox3 : waiting for response msg
>
> proxmox1 : waiting for response msg
>
> proxmox3 : waiting for response msg
>
> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging
>
> proxmox1 : given amount of query messages was sent
>
> proxmox3 : given amount of query messages was sent
>
> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040
>
> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041
>
> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023
>
> proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026
>
>
> Ok, so it seems we have a network problem on proxmox1 node. Network
> cards are as follows:
>
> - proxmox1: Intel X540 (10Gtek)
> - proxmox3: Intel X710 (Intel)
> - proxmox4: Intel X710 (Intel)
>
> Switch is Dell N1224T-ON.
>
> Does anyone have experience with Intel X540 chip network cards or Linux
> ixgbe network driver or 10Gtek manufacturer?
>
> If we change corosync communication to 1 Gbit network cards (broadcom)
> connected to an old HPE 1800-24G switch, cluster is stable...
>
> We also have a running cluster with Dell n1224T-ON switch and X710
> network cards without issues.
>
> Thanks a lot
> Eneko
>
>
>
>
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com<mailto:pve-user at pve.proxmox.com><mailto:pve-user at pve.proxmox.com>
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com<mailto:pve-user at pve.proxmox.com>
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
> --
> Zuzendari Teknikoa / Director Técnico
> Binovo IT Human Project, S.L.
> Telf. 943569206
> Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
> www.binovo.es<http://www.binovo.es/>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com<mailto:pve-user at pve.proxmox.com>
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
--
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943569206
Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
www.binovo.es
More information about the pve-user
mailing list