[PVE-User] PVE 3.4 to 4.1 upgrade terror history

[Eneko Lacunza]

Hi all,

This is a failed upgrade report from last friday.

We have a 4-node Proxmox cluster, with 3 of them running ceph MON and 3 
osd daemons each. All nodes are updated last PVE 3.4 :

node1: vms + ceph mon + 3xceph osd
node2: vms + ceph mon + 3xceph osd
node3: vms + ceph mon + 3xceph osd
node4: vms (actually 0 right now) + vzdumps (nfs)

To upgrade node2, we first moved all VMs running on "node2" to node1 & 
node3, then followed the wiki upgrade guide:
https://pve.proxmox.com/wiki/Upgrade_from_3.x_to_4.0

All was going quite well. We installed pve-kernel-4.2.2-1-pve and not 
the last available, which I think was the mistake.

The problem was that after "apt-get dist-upgrade" the server won't boot 
- it just kernel panic-ed. We tried booting with a PVE 3.4 kernel but 
the new userland (systemd) wasn't able to boot.

We tried various things (even installing the latest PVE kernel .deb with 
debian 8 rescue pendrives) but weren't able to fix it, so finally had to 
resinstall the server from scratch with the PVE 4.1 ISO.

Reinstalling with ISO was successful and we have even recovered the ceph 
OSD on that node.

This node2 server is a Dell T610, do you think the 4.2.2-1 kernel was 
faulty for this server? Maybe updating the wiki with the last PVE 4.1 
kernel could help others?

Maybe it could be a good idea to also install the debian 8 kernel, so 
that there is another option to boot in case the PVE kernel doesn't work?

Cheers
Eneko


-- 
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
       943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es