Multicast notes: Difference between revisions

From Proxmox VE
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
{{Note|Articles about Proxmox VE 2.0}}
== Introduction ==


= Introduction =
Multicast allows a single transmission to be delivered to multiple servers at the same time.
 
This is the basis for cluster communications in Proxmox VE 2.0 and higher, which uses corosync and cman, and would apply to any other solution which utilizes those clustering tools.
 
If multicast does not work in your network infrastructure, you should fix it so that it does.  If all else fails, use unicast instead, but beware of the node count limitations with unicast.
 
=== IGMP snooping ===
 
IGMP snooping prevents flooding multicast traffic to all ports in the broadcast domain by only allowing traffic destined for ports which have solicited such traffic.  IGMP snooping is a feature offered by most major switch manufacturers and is often enabled by default on switches.  In order for a switch to properly snoop the IGMP traffic, there must be an IGMP querier on the network.  If no querier is present, IGMP snooping will actively prevent ALL IGMP/Multicast traffic from being delivered!
 
If IGMP snooping is disabled, all multicast traffic will be delivered to all ports which may add unnecessary load, potentially allowing a denial of service attack.


Multicast allows a single transmission to be delivered to multiple servers at the same time.
=== IGMP querier ===


This is the basis for cluster communications in Proxmox VE 2.0.  
An IGMP querier is a multicast router that generates IGMP queries.  IGMP snooping relies on these queries which are unconditionally forwarded to all ports, as the replies from the destination ports is what builds the internal tables in the switch to allow it to know which traffic to forward.


If multicast does not work in your network infrastructure, use unicast instead.  
IGMP querier can be enabled on your router, switch, or even linux bridges.


= Troubleshooting =
== Troubleshooting ==


not all hosting companies allow multicast traffic.  
not all hosting companies allow multicast traffic.  
Line 15: Line 25:
Some switches have multicast disabled by default.  
Some switches have multicast disabled by default.  


== test if multicast is working between two nodes with omping==
=== test if multicast is working between two nodes with omping ===


  aptitude install omping
  aptitude install omping
Line 23: Line 33:
  omping node1 node2 node3
  omping node1 node2 node3


== test if multicast is working between two nodes with ssmping==
=== test if multicast is working between two nodes with ssmping ===


Copied from a post by e100 on forum .  
Copied from a post by e100 on forum .  
Line 66: Line 76:
  less /usr/share/doc/ssmping/README.gz
  less /usr/share/doc/ssmping/README.gz


== ssmping notes ==
=== ssmping notes ===


*there are a few other programs included in ssmping which may be of use. here is a list of the files in the package:
*there are a few other programs included in ssmping which may be of use. here is a list of the files in the package:
Line 90: Line 100:
then set up a cronjob to do ''apt-file update'' weekly or monthly ..
then set up a cronjob to do ''apt-file update'' weekly or monthly ..


== cman & iptables ==
=== cman & iptables ===
In case ''cman'' crashes with ''cpg_send_message failed: 9'' add those to your rule set:
In case ''cman'' crashes with ''cpg_send_message failed: 9'' add those to your rule set:
<pre>iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
<pre>iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
Line 96: Line 106:
</pre>
</pre>


= Use unicast instead of multicast =
== Use unicast instead of multicast ==


Unicast is a technology for sending messages to a single network destination. In corosync, unicast is implemented as UDP-unicast (UDPU). Due to increased network traffic (compared to multicast) the number of supported nodes is limited, do not use it with more that 4 cluster nodes.  
Unicast is a technology for sending messages to a single network destination. In corosync, unicast is implemented as UDP-unicast (UDPU). Due to increased network traffic (compared to multicast) the number of supported nodes is limited, do not use it with more that 4 cluster nodes.  
Line 110: Line 120:
*before you add a node, make sure you add all other nodes in /etc/hosts
*before you add a node, make sure you add all other nodes in /etc/hosts


= Multicast with Infiniband =
== Multicast with Infiniband ==


IP over Infiniband (IPoIB) supports Multicast but Multicast traffic is limited to 2044 Bytes when using connected mode even if you set a larger MTU on the IPoIB interface.  
IP over Infiniband (IPoIB) supports Multicast but Multicast traffic is limited to 2044 Bytes when using connected mode even if you set a larger MTU on the IPoIB interface.  
Line 116: Line 126:
Corosync has a setting, netmtu, that defaults to 1500 making it compatible with connected mode Infiniband.  
Corosync has a setting, netmtu, that defaults to 1500 making it compatible with connected mode Infiniband.  


== Changing netmtu ==
=== Changing netmtu ===


Changing the netmtu can increase throughput '''The following information is untested.'''  
Changing the netmtu can increase throughput '''The following information is untested.'''  
Line 141: Line 151:
<br>  
<br>  


= Netgear Managed Switches  =
== Disabling IGMP Snooping (not recommended) ==
 
=== Netgear Managed Switches  ===


the following are pics of setting to get multicast working on our netgear 7300 series switches. for more information see http://documentation.netgear.com/gs700at/enu/202-10360-01/GS700AT%20Series%20UG-06-18.html  
the following are pics of setting to get multicast working on our netgear 7300 series switches. for more information see http://documentation.netgear.com/gs700at/enu/202-10360-01/GS700AT%20Series%20UG-06-18.html  
Line 154: Line 166:




= Cisco Managed Switches  =
=== Cisco Managed Switches  ===


Some cisco switchs are a feature enabled by default : igmp snooping.
Some cisco switchs are a feature enabled by default : igmp snooping.

Revision as of 19:05, 13 October 2014

Introduction

Multicast allows a single transmission to be delivered to multiple servers at the same time.

This is the basis for cluster communications in Proxmox VE 2.0 and higher, which uses corosync and cman, and would apply to any other solution which utilizes those clustering tools.

If multicast does not work in your network infrastructure, you should fix it so that it does. If all else fails, use unicast instead, but beware of the node count limitations with unicast.

IGMP snooping

IGMP snooping prevents flooding multicast traffic to all ports in the broadcast domain by only allowing traffic destined for ports which have solicited such traffic. IGMP snooping is a feature offered by most major switch manufacturers and is often enabled by default on switches. In order for a switch to properly snoop the IGMP traffic, there must be an IGMP querier on the network. If no querier is present, IGMP snooping will actively prevent ALL IGMP/Multicast traffic from being delivered!

If IGMP snooping is disabled, all multicast traffic will be delivered to all ports which may add unnecessary load, potentially allowing a denial of service attack.

IGMP querier

An IGMP querier is a multicast router that generates IGMP queries. IGMP snooping relies on these queries which are unconditionally forwarded to all ports, as the replies from the destination ports is what builds the internal tables in the switch to allow it to know which traffic to forward.

IGMP querier can be enabled on your router, switch, or even linux bridges.

Troubleshooting

not all hosting companies allow multicast traffic.

Some switches have multicast disabled by default.

test if multicast is working between two nodes with omping

aptitude install omping

start omping on all nodes with the following command and check the output, e.g: 

omping node1 node2 node3

test if multicast is working between two nodes with ssmping

Copied from a post by e100 on forum .

  • this uses ssmping

Install this on all nodes . 

aptitude install ssmping

run this on Node A: 

ssmpingd

then on Node B: 

asmping 224.0.2.1 ip_for_NODE_A_here

example output 

asmping joined (S,G) = (*,224.0.2.234)
pinging 192.168.8.6 from 192.168.8.5
  unicast from 192.168.8.6, seq=1 dist=0 time=0.221 ms
  unicast from 192.168.8.6, seq=2 dist=0 time=0.229 ms
multicast from 192.168.8.6, seq=2 dist=0 time=0.261 ms
  unicast from 192.168.8.6, seq=3 dist=0 time=0.198 ms
multicast from 192.168.8.6, seq=3 dist=0 time=0.213 ms
  unicast from 192.168.8.6, seq=4 dist=0 time=0.234 ms
multicast from 192.168.8.6, seq=4 dist=0 time=0.248 ms
  unicast from 192.168.8.6, seq=5 dist=0 time=0.249 ms
multicast from 192.168.8.6, seq=5 dist=0 time=0.263 ms
  unicast from 192.168.8.6, seq=6 dist=0 time=0.250 ms
multicast from 192.168.8.6, seq=6 dist=0 time=0.264 ms
  unicast from 192.168.8.6, seq=7 dist=0 time=0.245 ms
multicast from 192.168.8.6, seq=7 dist=0 time=0.260 ms

for more information see 

man ssmping

and 

less /usr/share/doc/ssmping/README.gz

ssmping notes

  • there are a few other programs included in ssmping which may be of use. here is a list of the files in the package:
apt-file list ssmping

ssmping: /usr/bin/asmping
ssmping: /usr/bin/mcfirst
ssmping: /usr/bin/ssmping
ssmping: /usr/bin/ssmpingd
ssmping: /usr/share/doc/ssmping/README.gz
ssmping: /usr/share/doc/ssmping/changelog.Debian.gz
ssmping: /usr/share/doc/ssmping/copyright
ssmping: /usr/share/man/man1/asmping.1.gz
ssmping: /usr/share/man/man1/mcfirst.1.gz
ssmping: /usr/share/man/man1/ssmping.1.gz
ssmping: /usr/share/man/man1/ssmpingd.1.gz
  • If you want to use apt-file do this:
aptitude install apt-file
apt-file update

then set up a cronjob to do apt-file update weekly or monthly ..

cman & iptables

In case cman crashes with cpg_send_message failed: 9 add those to your rule set: 

iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
iptables -A INPUT -p udp -m state --state NEW -m multiport –dports 5404,5405 -j ACCEPT

Use unicast instead of multicast

Unicast is a technology for sending messages to a single network destination. In corosync, unicast is implemented as UDP-unicast (UDPU). Due to increased network traffic (compared to multicast) the number of supported nodes is limited, do not use it with more that 4 cluster nodes. 

<cman keyfile="/var/lib/pve-cluster/corosync.authkey" transport="udpu"/>
  • activate via GUI
  • add all nodes you want to join in /etc/hosts and reboot
  • before you add a node, make sure you add all other nodes in /etc/hosts

Multicast with Infiniband

IP over Infiniband (IPoIB) supports Multicast but Multicast traffic is limited to 2044 Bytes when using connected mode even if you set a larger MTU on the IPoIB interface.

Corosync has a setting, netmtu, that defaults to 1500 making it compatible with connected mode Infiniband.

Changing netmtu

Changing the netmtu can increase throughput The following information is untested.

Edit the /etc/pve/cluster.conf file Add the section: 

<totem netmtu="2044" />



<?xml version="1.0"?>
<cluster name="clustername" config_version="2">
  <totem netmtu="2044" />
  <cman keyfile="/var/lib/pve-cluster/corosync.authkey">
  </cman>

  <clusternodes>
  <clusternode name="node1" votes="1" nodeid="1"/>
  <clusternode name="node2" votes="1" nodeid="2"/>
  <clusternode name="node3" votes="1" nodeid="3"/></clusternodes>

</cluster>


Disabling IGMP Snooping (not recommended)

Netgear Managed Switches

the following are pics of setting to get multicast working on our netgear 7300 series switches. for more information see http://documentation.netgear.com/gs700at/enu/202-10360-01/GS700AT%20Series%20UG-06-18.html


Multicast-netgear-1.png

Multicast-netgear-2.png

Multicast-netgear-3.png

NetGear-multicast-save-and-apply.png


Cisco Managed Switches

Some cisco switchs are a feature enabled by default : igmp snooping.

These feature is used to filter multicast traffic, to avoid to forward it on each ports.

But this can sometimes do problems with corosync, so it's better to disable it.


For cisco 2960G, by example, you can disable it with: 

# conf t
# no ip igmp snooping
Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Multicast_notes&oldid=6786"