[PVE-User] Proxmox VE 3.0 RC2 -- crashed.. containers will not start .. possible kernel error

[admin extremeshok.com]

Hi Guys

Upgraded to 3.0 rc2 last night, everything was working fine.

This morning the containers are all offline.

According the the root console the containers are running. I am usable
to shutdown/stop/start the containers via root console or webconsole.

Really need your guys help, as I have no idea where to even start
looking for the problem.

Thanks

root at prox:~# uname -a
Linux prox 2.6.32-20-pve #1 SMP Wed May 15 08:23:27 CEST 2013 x86_64
GNU/Linux


root at prox:~# tail -f /var/log/kern.log
May 17 02:55:17 prox kernel: [<ffffffffa00fe4b4>]
ext4_page_mkwrite+0x74/0x3b0 [ext4]
May 17 02:55:17 prox kernel: [<ffffffff81151705>] do_wp_page+0x775/0x960
May 17 02:55:17 prox kernel: [<ffffffff81099317>] ?
hrtimer_try_to_cancel+0x47/0xd0
May 17 02:55:17 prox kernel: [<ffffffff8115631d>]
handle_pte_fault+0x53d/0x17a0
May 17 02:55:17 prox kernel: [<ffffffff810b22ca>] ?
get_futex_key+0x15a/0x280
May 17 02:55:17 prox kernel: [<ffffffff811577b9>]
handle_mm_fault+0x239/0x310
May 17 02:55:17 prox kernel: [<ffffffff81040471>]
__do_page_fault+0x181/0x490
May 17 02:55:17 prox kernel: [<ffffffff8151ad3c>] ? thread_return+0xbc/0x870
May 17 02:55:17 prox kernel: [<ffffffff81520b7b>] do_page_fault+0x3b/0xa0
May 17 02:55:17 prox kernel: [<ffffffff8151df05>] page_fault+0x25/0x30


root at prox:~# pvectl list
      VMID NAME                 STATUS     MEM(MB)    DISK(GB)   
       104 dev.xshok.com        running    8192       100.00     
       106 my.mailshok.com      running    16384      500.00     
       107 archive.mailshok.com running    2048       500.00     
       110 cobalt.xshok.com     running    4096       20.00      
       111 cyan.xshok.com       running    8192       50.00      
       112 chestnut.xshok.com   running    8192       100.00     
       113 copper.xshok.com     running    16384      100.00     
       114 coral.xshok.com      running    16384      100.00     
root at prox:~#

root at prox:~# tail -f /var/log/vzctl.log
2013-05-16T20:13:32-0500 vzctl : CT 113 : Setting CPUs: 8
2013-05-16T20:13:32-0500 vzctl : CT 113 : Container start in progress...
2013-05-16T20:13:32-0500 vzctl : CT 114 : Starting container ...
2013-05-16T20:13:35-0500 vzctl : CT 114 : Container is mounted
2013-05-16T20:13:35-0500 vzctl : CT 114 : Adding IP address(es):
68.235.32.14
2013-05-16T20:13:37-0500 vzctl : CT 114 : Setting CPU units: 100000
2013-05-16T20:13:37-0500 vzctl : CT 114 : Setting CPUs: 8
2013-05-16T20:13:37-0500 vzctl : CT 114 : Container start in progress...
2013-05-17T03:42:28-0500 vzctl : CT 104 : Killing container ...
2013-05-17T03:43:28-0500 vzctl : CT 104 : Unable to stop container:
operation timed out

root at prox:~# pveversion -v
pve-manager: 3.0-17 (pve-manager/3.0/f8f25665)
running kernel: 2.6.32-20-pve
proxmox-ve-2.6.32: 3.0-100
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-19-pve: 2.6.32-96
lvm2: 2.02.95-pve3
clvm: 2.02.95-pve3
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-2
qemu-server: 3.0-12
pve-firmware: 1.0-22
libpve-common-perl: 3.0-4
libpve-access-control: 3.0-4
libpve-storage-perl: 3.0-6
vncterm: 1.1-3
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-12
ksm-control-daemon: 1.1-1