[PVE-User] Ceph over IP over Infiniband

[Vadim Bulst]

Hi Phil,

I also use an Infiniband network for a Ceph cluster. IPoverIB should 
work. My loaded modules:

mlx4_ib
rdma_ucm
ib_umad
ib_uverbs
ib_ipoib
ib_srp
ib_sdp

Your /etc/network/interfaces should be similar to that:

auto ib0
iface ib0 inet static
                  address 10.10.144.212
                  netmask 255.255.255.0
                  mtu 65520
                  pre-up echo connected > /sys/class/net/ib0/mode


This configuration works with Mellanox cards and switches.


Cheers,

Vadim


On 12/19/2017 12:28 PM, Phil Schwarz wrote:
> Hi,
> I'm currently trying to set up a brand new home cluster :
> - 5 nodes, with each :
>
> - 1 HCA Mellanox ConnectX-2
> - 1 GB Ethernet (Proxmox 5.1 Network Admin)
> - 1 CX4 to CX4 cable
>
> All together connected to a SDR Flextronics IB Switch.
>
> This setup should back a Ceph Luminous (V12.2.2 included in proxmox
> V5.1) On all nodes, I did:
> - apt-get infiniband-diags
> - modprobe mlx4_ib
> - modprobe ib_ipoib
> - modprobe ib_umad
> - ifconfig ib0 IP/MASK
>
> On two nodes (tried previously on a single on, same issue), i installed
> opensm ( The switch doesn't have SM included) :
> apt-get install opensm
> /etc/init.d/opensm stop
> /etc/init.d/opensm start
> (Necessary to let the daemon create the logfiles)
>
> I tailed the logfile and got a "Active&Running" Setup, with "SUBNET UP"
>
> Every node is OK regardless to IB Setup :
> - All ib0 are UP, using ibstat
> - ibhosts and ibswitches seem to be OK
>
> On a node :
> ibping -S
>
> On every other node :
> ibping -G GID_Of_Previous_Server_Port
>
> I got a nice pong reply on every node. Should be happy, but...
> But i never went further.. Tried to ping each other. No way to get into
> this (mostly probably) simple issue...
>
>
> Any hint to achieve this task ??
>
>
> Thanks for all
> Best regards
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

-- 
Vadim Bulst

Universität Leipzig / URZ
04109  Leipzig, Augustusplatz 10

phone: ++49-341-97-33380
mail:    vadim.bulst at uni-leipzig.de