[PVE-User] LV Not Available - could not open disk - no such file or directory

[John Crisp]

Earlier today we had a working Prox 4.2.2 installation

There was an issue with the BBWC on the RAID that was fixed by the data
centre.

After we restarted the machine (and apparent successful shutdown of the
VM) we found that the VMs failed to start with the type of error as
detailed below.

The machine was a 'default' install of Prox with lvm-thin

Each of the 3 VMs shows the same type of error with "Could not open
....disk-1 No such file or directory"

AFAWAA nothing had been touched with respect to the LVMs

The link for the 'disk' does not appear in /dev/pve (though I believe I
saw one in there prior to the machine being restarted)

file=/dev/pve/vm-701-disk-1


root at pve:~# ls -l /dev/pve
total 0
lrwxrwxrwx 1 root root 7 Sep  1 02:25 data_meta0 -> ../dm-3
lrwxrwxrwx 1 root root 7 Sep  1 02:24 root -> ../dm-0
lrwxrwxrwx 1 root root 7 Sep  1 02:25 swap -> ../dm-1
lrwxrwxrwx 1 root root 7 Sep  1 02:25 temp -> ../dm-2

We have applied the latest updates from the pve-no-subscription list

proxmox-ve                     4.2-60

Frustratingly if we look in the web panel under local-lvm we can see the
Disk Images..... but just cannot access them !

We also tried to create a test disk on another VM and got this error

lvcreate 'pve/vm-201-disk-2' error: Failed to activate thin pool
pve/data. (500)

That gave us a clue to look at lvdisplay and vgdisplay and we can see
that both the main LVM and the disks show as 'Not Available' (see below)

We presume there has been some corruption somewhere.

>From what we can see we could try something like these commands but
don't want to make matters worse....

lvchange -a y /dev/pve

vgscan --mknodes

vgchange -a y


Any help grateful received

B. Rgds
John

  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                nEYeJ4-J9R9-cWBm-GMMo-AkRr-4kqx-t7WsuW
  LV Write Access        read/write
  LV Creation host, time proxmox, 2016-07-22 20:33:31 +0200
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              NOT available
  LV Size                5.34 TiB
  Current LE             1399264
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

  --- Logical volume ---
  LV Path                /dev/pve/vm-701-disk-1
  LV Name                vm-701-disk-1
  VG Name                pve
  LV UUID                cRNUxc-ve7M-4Q60-Hpji-XaQr-n3PT-Gqhhs4
  LV Write Access        read/write
  LV Creation host, time pve, 2016-07-22 22:32:42 +0200
  LV Pool name           data
  LV Status              NOT available
  LV Size                1000.00 GiB
  Current LE             256000
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

root at pve:/etc/pve/nodes/pve/qemu-server# vgdisplay
  --- Volume group ---
  VG Name               pve
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  19
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                8
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.45 TiB
  PE Size               4.00 MiB
  Total PE              1429983
  Alloc PE / Size       1425934 / 5.44 TiB
  Free  PE / Size       4049 / 15.82 GiB
  VG UUID               y2MaBy-hNxj-f7yX-DJRC-dAE7-udMJ-rDVybN


root at pve:/etc/pve/nodes/pve/qemu-server# lvs
LV            VG   Attr       LSize    Pool Origin Data%  Meta%  Move
Log Cpy%Sync Convert
data          pve  twi---tz--    5.34t
data_meta0    pve  -wi-a-----    4.00m
root          pve  -wi-ao----   96.00g
swap          pve  -wi-ao----    8.00g
temp          pve  -wi-a----    88.00m
vm-200-disk-1 pve  Vwi---tz--  100.00g data
vm-201-disk-1 pve  Vwi---tz--   32.00g data
vm-701-disk-1 pve  Vwi---tz-- 1000.00g data



Running as unit 701.scope.

kvm: -drive
file=/dev/pve/vm-701-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on:
Could not open '/dev/pve/vm-701-disk-1': No such file or directory

TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice
qemu --unit 701 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id
701 -chardev
'socket,id=qmp,path=/var/run/qemu-server/701.qmp,server,nowait' -mon
'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/701.pid
-daemonize -smbios 'type=1,uuid=6e58224b-84a4-4298-b3c4-c325c055fdee'
-name worldroadstatistics.ch -smp '8,sockets=2,cores=4,maxcpus=8'
-nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus
-vnc unix:/var/run/qemu-server/701.vnc,x509,password -cpu
kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -object
'memory-backend-ram,size=2048M,id=ram-node0' -numa
'node,nodeid=0,cpus=0-3,memdev=ram-node0' -object
'memory-backend-ram,size=2048M,id=ram-node1' -numa
'node,nodeid=1,cpus=4-7,memdev=ram-node1' -k en-us -device
'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device
'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device
'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device
'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi
'initiator-name=iqn.1993-08.org.debian:01:7b514af6ea7c' -drive
'file=/var/lib/vz/template/iso/smeserver-9.1-x86_64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads'
-device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200'
-drive
'file=/dev/pve/vm-701-disk-1,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on'
-device
'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100'
-netdev
'type=tap,id=net0,ifname=tap701i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on'
-device
'virtio-net-pci,mac=36:32:33:36:31:35,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300''
failed: exit code 1