[PVE-User] Cluster disaster

[Dhaussy Alexandre]

> Do you have a hint why there is no messages in the logs when watchdog
> actually seems to trigger fencing ?
> Because when a node suddently reboots, i can't be sure if it's the watchdog,
> a hardware bug, kernel bug or whatever..

Responding to myself, i find this interesting :

Nov  8 10:39:01 proxmoxt35 corosync[35250]:  [TOTEM ] A new membership (10.xx.xx.11:684) was formed. Members joined: 13
Nov  8 10:39:58 proxmoxt35 watchdog-mux[28239]: client watchdog expired - disable watchdog updates

Nov  8 10:39:01 proxmoxt31 corosync[23483]:  [TOTEM ] A new membership (10.xx.xx.11:684) was formed. Members joined: 13
Nov  8 10:40:01 proxmoxt31 watchdog-mux[22395]: client watchdog expired - disable watchdog updates

Nov  8 10:39:01 proxmoxt30 corosync[24634]:  [TOTEM ] A new membership (10.xx.xx.11:684) was formed. Members joined: 13
Nov  8 10:40:00 proxmoxt30 watchdog-mux[23492]: client watchdog expired - disable watchdog updates


Nov  9 10:05:41 proxmoxt20 corosync[42543]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt20 corosync[42543]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt20 watchdog-mux[41401]: client watchdog expired - disable watchdog updates

Nov  9 10:05:41 proxmoxt21 corosync[16184]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt21 corosync[16184]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt21 watchdog-mux[42853]: client watchdog expired - disable watchdog updates

Nov  9 10:05:41 proxmoxt30 corosync[16159]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt30 corosync[16159]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt30 watchdog-mux[43148]: client watchdog expired - disable watchdog updates

Nov  9 10:05:41 proxmoxt31 corosync[16297]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt31 corosync[16297]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt31 watchdog-mux[42761]: client watchdog expired - disable watchdog updates

Nov  9 10:05:41 proxmoxt34 corosync[41330]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt34 corosync[41330]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt34 watchdog-mux[40262]: client watchdog expired - disable watchdog updates

Nov  9 10:05:41 proxmoxt35 corosync[16158]:  [TOTEM ] A new membership (10.xx.xx.11:796) was formed. Members left: 7
Nov  9 10:05:46 proxmoxt35 corosync[16158]:  [TOTEM ] A new membership (10.xx.xx.11:800) was formed. Members joined: 7
Nov  9 10:06:42 proxmoxt35 watchdog-mux[42684]: client watchdog expired - disable watchdog updates