[PVE-User] Proxmox Kernel / Ceph Integration

[Thomas Lamprecht]

Hi,

Am 07/27/2018 um 11:02 AM schrieb Marcus Haarmann:
> Hi experts,
> 
> we are using a Proxmox cluster with an underlying ceph storage.
> Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous 12.2.5
> We are running a couple of VM and also Containers there.
> 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit interfaces (public/cluster separated)
> 
> It occurs during nightly backup that backup stalls. In parallel, we get lots of messages in the dmesg:
> [137612.371311] libceph: mon0 192.168.16.31:6789 session established
> [137643.090541] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon
> [137643.091383] libceph: mon1 192.168.16.32:6789 session established
> [137673.810526] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon
> [137673.811388] libceph: mon2 192.168.16.34:6789 session established
> [137704.530567] libceph: mon2 192.168.16.34:6789 session lost, hunting for new mon
> [137704.531363] libceph: mon0 192.168.16.31:6789 session established
> [137735.250593] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon
> [137735.251352] libceph: mon1 192.168.16.32:6789 session established
> [137765.970608] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon
> [137765.971544] libceph: mon0 192.168.16.31:6789 session established
> [137796.690605] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon
> [137796.691412] libceph: mon1 192.168.16.32:6789 session established
> 
> We are searching for the issue for a while, since the blocking backup is not easy to overcome (unblocking does not help,
> only stop and migrate to a different server, since the rbd device seems to block).
> It seems to be related to the ceph messages.
> We found the following patch related to these messages (which may lead to a blocking state in kernel):
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7b4c443d139f1d2b5570da475f7a9cbcef86740c
> 
> We have tried to patch the kernel ourselfes, but this was not successful.
> 

Porting the patch was not successful or the patch did not worked as
expected?

> Although I presume the real error situation is related to a network problem, it would be nice to have an
> official backport of this patch in the pve kernel.
> Anybody can do that ? (only one line of code)
> 

A single line can also wreak havoc just fine ;-)
But this one seems/sounds harmless, regression-wise. But it would be
really good to first know if the patch addresses the issue at all.

> We are trying to modify the bonding mode because the network connection seems to be unstable,
> maybe this solves the issue.
> 

Sounds like it's worth a shot, if you already know that the network may
not be fully stable, as you may want to do something about that sooner
or later anyway.

cheers,
Thomas