Full Mesh Network for Ceph Server

From Proxmox VE
Jump to navigation Jump to search

Introduction

This wiki page describes how to configure in Proxmox VE (or any other Debian based LINUX distribution) a three node "Meshed Network" (instead of a network switch) as it can be used e.g. for connecting Ceph Servers or nodes in a Proxmox VE Cluster. This should also work with a 5-node cluster, general you need nodes_total - 1 = nic ports. The basic idea is running a small 3 node cluster with 10 Gbit network WITHOUT buying an expensive 10 Gbit network switch.

This should work with any kind of ethernet NICs, i.e. also 40 Gbit or even 100 Gbit ones. But for verifying this article, 10 Gbit Intel NICs were used.

There a two possible method to achieve a full mesh. General the fist one is the recommended one, because it is easier to set up and supports multicast.

Method 1

Create a "broadcast" bond with the given interfaces on every node. This can be done over the GUI or on the command-line.

GUI

On the GUI go to the node level -> System -> Network. Then click on "Create" and select "Linux Bond". In the Wizard make your configuration without a gateway and set mode to "broadcast".

Reboot the node to activate the new network settings.

Command-Line

Add the following lines to '/etc/network/interfaces'. 

auto bond<No>
iface bond<No> inet static
	address  <IP>
	netmask  <Netmask>
	slaves <Nic1> <Nic2>
	bond_miimon 100
	bond_mode broadcast
#Full Mesh

Then start the bond 

ifup bond<No>

Method 2

Node1

/etc/network/interface

auto lo
iface lo inet loopback

iface eth2 inet manual

auto eth3
iface eth3 inet static
        address  10.14.14.50
        netmask  255.255.255.0

# Connected to Node2 (.51)
auto eth0
iface eth0 inet static
        address  10.15.15.50
        netmask  255.255.255.0
        up route add -net 10.15.15.51 netmask 255.255.255.255 dev eth0
        down route del -net 10.15.15.51 netmask 255.255.255.255 dev eth0

# Connected to Node3 (.52)
auto eth1
iface eth1 inet static
        address  10.15.15.50
        netmask  255.255.255.0
        up route add -net 10.15.15.52 netmask 255.255.255.255 dev eth1
        down route del -net 10.15.15.52 netmask 255.255.255.255 dev eth1

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.50
        netmask  255.255.240.0
        gateway  192.168.2.1
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0

route

root@pve-2-50:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.15.15.51     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.15.15.52     0.0.0.0         255.255.255.255 UH    0      0        0 eth1
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.14.14.0      0.0.0.0         255.255.255.0   U     0      0        0 eth3
192.168.0.0     0.0.0.0         255.255.240.0   U     0      0        0 vmbr0
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 vmbr0

Node2

/etc/network/interface

auto lo
iface lo inet loopback

iface eth2 inet manual

auto eth3
iface eth3 inet static
        address  10.14.14.51
        netmask  255.255.255.0

# Connected to Node1 (.50)
auto eth0
iface eth0 inet static
        address  10.15.15.51
        netmask  255.255.255.0
        up route add -net 10.15.15.50 netmask 255.255.255.255 dev eth0
        down route del -net 10.15.15.50 netmask 255.255.255.255 dev eth0

# Connected to Node3 (.52)
auto eth1
iface eth1 inet static
        address  10.15.15.51
        netmask  255.255.255.0
        up route add -net 10.15.15.52 netmask 255.255.255.255 dev eth1
        down route del -net 10.15.15.52 netmask 255.255.255.255 dev eth1

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.51
        netmask  255.255.240.0
        gateway  192.168.2.1
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0

route

root@pve-2-51:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.15.15.50     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.15.15.52     0.0.0.0         255.255.255.255 UH    0      0        0 eth1
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.14.14.0      0.0.0.0         255.255.255.0   U     0      0        0 eth3
192.168.0.0     0.0.0.0         255.255.240.0   U     0      0        0 vmbr0
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 vmbr0

Node3

/etc/network/interface

auto lo
iface lo inet loopback

iface eth2 inet manual

auto eth3
iface eth3 inet static
        address  10.14.14.52
        netmask  255.255.255.0

# Connected to Node2 (.51)
auto eth0
iface eth0 inet static
        address  10.15.15.52
        netmask  255.255.255.0
        up route add -net 10.15.15.51 netmask 255.255.255.255 dev eth0
        down route del -net 10.15.15.51 netmask 255.255.255.255 dev eth0

# Connected to Node1 (.50)
auto eth1
iface eth1 inet static
        address  10.15.15.52
        netmask  255.255.255.0
        up route add -net 10.15.15.50 netmask 255.255.255.255 dev eth1
        down route del -net 10.15.15.50 netmask 255.255.255.255 dev eth1

auto vmbr0
iface vmbr0 inet static
        address  192.168.2.52
        netmask  255.255.240.0
        gateway  192.168.2.1
        bridge_ports eth2
        bridge_stp off
        bridge_fd 0

route

root@pve-2-52:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.15.15.51     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
10.15.15.50     0.0.0.0         255.255.255.255 UH    0      0        0 eth1
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0
10.15.15.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
10.14.14.0      0.0.0.0         255.255.255.0   U     0      0        0 eth3
192.168.0.0     0.0.0.0         255.255.240.0   U     0      0        0 vmbr0
0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 vmbr0
Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Full_Mesh_Network_for_Ceph_Server&oldid=9651"