[PVE-User] LVM autoactivation failed with multipath over iSCSI
nada
nada at verdnatura.es
Wed Jan 15 13:16:29 CET 2020
thank you Stefan for your tips !!
i do not do activation manualy, i use rc-local.service (details follows)
but i will reconfigure multipath in the evening and reboot both nodes
will let you know results by tomorrow
have a nice day ;-)
Nada
root at mox11:~# cat
/etc/systemd/system/multi-user.target.wants/rc-local.service
[Unit]
Description=/etc/rc.local Compatibility
Documentation=man:systemd-rc-local-generator(8)
ConditionFileIsExecutable=/etc/rc.local
After=network.target iscsid.service multipathd.service
open-iscsi.service
[Service]
Type=forking
ExecStart=/etc/rc.local
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no
[Install]
WantedBy=multi-user.target
root at mox11:~# cat /etc/rc.local
#!/bin/bash
# just to activate VGs from SAN
/bin/sleep 10
/sbin/vgchange -aly santest
/sbin/vgchange -aly san2020jan
On 2020-01-15 12:06, Stefan M. Radman wrote:
> Hi Nada,
>
> Unfortunately I don't have any first hand experience with PVE6 yet.
>
> On the PVE5.4 cluster I am working with I had an issue that looked
> very similar to yours:
> LVM refused to activate iSCSI multipath volumes on boot, making the
> lvm2-activation-net.service fail.
> This only happened during boot of the host.
> Restarting the lvm2-activation-net.service after boot activated the
> volume with multipath working.
> Suspecting a timing/dependency issue specific to my configuration I
> took a pragmatic approach and added a custom systemd service template
> to restart the lvm2-activation-net.service after multipath
> initialization (see below).
>
> # cat /etc/systemd/system/lvm2-after-multipath.service
> [Unit]
> Description=LVM2 after Multipath
> After=multipathd.service lvm2-activation-net.service
>
> [Service]
> Type=oneshot
> ExecStart=/bin/systemctl start lvm2-activation-net.service
>
> [Install]
> WantedBy=sysinit.target
>
> Things on PVE6 seem to have changed a bit but your lvm2-pvescan
> service failures indicate a similar problem ("failed LVM event
> activation").
> Disable your rc.local workaround and try to restart the two failed
> services after reboot.
> If that works you might want to take a similar approach instead of
> activating the volumes manually.
>
> The masked status of multipath-tools-boot.service is ok. The package
> is only needed when booting from multipath devices.
>
> Your mistake in multipath.conf seems to be in the multipaths section.
> Each multipath device can only have one WWID. For two WWIDs you'll
> need two multiparty subsections.
> https://help.ubuntu.com/lts/serverguide/multipath-dm-multipath-config-file.html#multipath-config-multipath
>
> Stefan
>
> On Jan 15, 2020, at 10:55, nada
> <nada at verdnatura.es<mailto:nada at verdnatura.es>> wrote:
>
> On 2020-01-14 19:46, Stefan M. Radman via pve-user wrote:
> Hi Nada
> What's the output of "systemctl --failed" and "systemctl status
> lvm2-activation-net.service".
> Stefan
>
> Hi Stefan
> thank you for your response !
> the output of "systemctl --failed" was claiming devices from SAN during
> boot,
> which were activated by rc-local.service after boot
> i do NOT have "lvm2-activation-net.service"
> and find masked status of multipath-tools-boot.service < is this ok ???
> but i find some mistake in multipath.conf eventhough it is running
> i am going to reconfigure it and reboot this afternoon
> following are details
> Nada
>
> root at mox11:~# systemctl --failed --all
> UNIT LOAD ACTIVE SUB DESCRIPTION
> lvm2-pvscan at 253:7.service loaded failed failed LVM event activation
> on device 253:7
> lvm2-pvscan at 253:8.service loaded failed failed LVM event activation
> on device 253:8
>
> LOAD = Reflects whether the unit definition was properly loaded.
> ACTIVE = The high-level unit activation state, i.e. generalization of
> SUB.
> SUB = The low-level unit activation state, values depend on unit
> type.
>
> 2 loaded units listed.
> To show all installed unit files use 'systemctl list-unit-files'.
>
> root at mox11:~# dmsetup ls
> san2020jan-vm--903--disk--0 (253:18)
> santest-santestpool (253:12)
> 3600c0ff000195f8e7d0a855701000000 (253:7)
> pve-data-tpool (253:4)
> pve-data_tdata (253:3)
> pve-zfs (253:6)
> santest-santestpool-tpool (253:11)
> santest-santestpool_tdata (253:10)
> pve-data_tmeta (253:2)
> san2020jan-san2020janpool (253:17)
> santest-santestpool_tmeta (253:9)
> pve-swap (253:0)
> pve-root (253:1)
> pve-data (253:5)
> 3600c0ff000195f8ec3f01d5e01000000 (253:8)
> san2020jan-san2020janpool-tpool (253:16)
> san2020jan-san2020janpool_tdata (253:15)
> san2020jan-san2020janpool_tmeta (253:14)
>
> root at mox11:~# pvs -a
> PV VG Fmt Attr
> PSize PFree
> /dev/mapper/3600c0ff000195f8e7d0a855701000000 santest lvm2 a--
> <9.31g 292.00m
> /dev/mapper/3600c0ff000195f8ec3f01d5e01000000 san2020jan lvm2 a--
> <93.13g <2.95g
> /dev/mapper/san2020jan-vm--903--disk--0 ---
> 0 0
> /dev/sdb ---
> 0 0
> /dev/sdc2 ---
> 0 0
> /dev/sdc3 pve lvm2 a--
> 67.73g 6.97g
> root at mox11:~# vgs -a
> VG #PV #LV #SN Attr VSize VFree
> pve 1 4 0 wz--n- 67.73g 6.97g
> san2020jan 1 2 0 wz--n- <93.13g <2.95g
> santest 1 1 0 wz--n- <9.31g 292.00m
> root at mox11:~# lvs -a
> LV VG Attr LSize Pool
> Origin Data% Meta% Move Log Cpy%Sync Convert
> data pve twi-aotz-- 9.98g
> 0.00 10.61
> [data_tdata] pve Twi-ao---- 9.98g
> [data_tmeta] pve ewi-ao---- 12.00m
> [lvol0_pmspare] pve ewi------- 12.00m
> root pve -wi-ao---- 16.75g
> swap pve -wi-ao---- 4.00g
> zfs pve -wi-ao---- 30.00g
> [lvol0_pmspare] san2020jan ewi------- 92.00m
> san2020janpool san2020jan twi-aotz-- 90.00g
> 0.86 10.84
> [san2020janpool_tdata] san2020jan Twi-ao---- 90.00g
> [san2020janpool_tmeta] san2020jan ewi-ao---- 92.00m
> vm-903-disk-0 san2020jan Vwi-aotz-- 2.50g san2020janpool
> 30.99
> [lvol0_pmspare] santest ewi------- 12.00m
> santestpool santest twi-aotz-- 9.00g
> 0.00 10.58
> [santestpool_tdata] santest Twi-ao---- 9.00g
> [santestpool_tmeta] santest ewi-ao---- 12.00m
>
> root at mox11:~# multipathd -k"show maps"
> Jan 15 10:50:02 | /etc/multipath.conf line 24, duplicate keyword: wwid
> name sysfs uuid
> 3600c0ff000195f8e7d0a855701000000 dm-7
> 3600c0ff000195f8e7d0a855701000000
> 3600c0ff000195f8ec3f01d5e01000000 dm-8
> 3600c0ff000195f8ec3f01d5e01000000
>
> root at mox11:~# multipathd -k"show paths"
> Jan 15 10:50:07 | /etc/multipath.conf line 24, duplicate keyword: wwid
> hcil dev dev_t pri dm_st chk_st dev_st next_check
> 6:0:0:3 sde 8:64 10 active ready running XX........ 2/8
> 9:0:0:3 sdn 8:208 10 active ready running XX........ 2/8
> 7:0:0:3 sdh 8:112 50 active ready running XX........ 2/8
> 5:0:0:3 sdd 8:48 10 active ready running XXXXXX.... 5/8
> 11:0:0:3 sdp 8:240 50 active ready running X......... 1/8
> 10:0:0:3 sdl 8:176 50 active ready running XXXXXXXXXX 8/8
> 8:0:0:6 sdk 8:160 50 active ready running XXXXXX.... 5/8
> 8:0:0:3 sdj 8:144 50 active ready running XXXXXXXXXX 8/8
> 9:0:0:6 sdo 8:224 10 active ready running X......... 1/8
> 6:0:0:6 sdg 8:96 10 active ready running XXXXXX.... 5/8
> 5:0:0:6 sdf 8:80 10 active ready running XXXXXX.... 5/8
> 10:0:0:6 sdm 8:192 50 active ready running XXXXXXX... 6/8
> 11:0:0:6 sdq 65:0 50 active ready running XXXXXXXX.. 7/8
> 7:0:0:6 sdi 8:128 50 active ready running XXXXXXX... 6/8
>
> root at mox11:~# cat /etc/multipath.conf
> defaults {
> polling_interval 2
> uid_attribute ID_SERIAL
> no_path_retry queue
> find_multipaths yes
> }
> blacklist {
> wwid .*
> # BECAREFULL @mox11 blacklit sda,sdb,sdc
> devnode "^sd[a-c]"
> }
> blacklist_exceptions {
> # 25G v_multitest
> # wwid "3600c0ff000195f8e2172de5d01000000"
> # 10G prueba
> wwid "3600c0ff000195f8e7d0a855701000000"
> # 100G sanmox11
> wwid "3600c0ff000195f8ec3f01d5e01000000"
> }
> multipaths {
> multipath {
> # wwid "3600c0ff000195f8e2172de5d01000000"
> wwid "3600c0ff000195f8e7d0a855701000000"
> wwid "3600c0ff000195f8ec3f01d5e01000000"
> }
> }
> devices {
> device {
> #### the following 6 lines do NOT change
> vendor "HP"
> product "P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI"
> # getuid_callout "/lib/udev/scsi_id -g -u -s /block/%n"
> path_grouping_policy "group_by_prio"
> prio "alua"
> failback "immediate"
> no_path_retry 18
> ####
> hardware_handler "0"
> path_selector "round-robin 0"
> rr_weight uniform
> rr_min_io 100
> path_checker tur
> }
> }
>
>
> root at mox11:~# systemctl status multipath-tools-boot
> multipath-tools-boot.service
> Loaded: masked (Reason: Unit multipath-tools-boot.service is masked.)
> Active: inactive (dead)
>
> root at mox11:~# pveversion -V
> proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
> pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
> pve-kernel-5.3: 6.1-1
> pve-kernel-helper: 6.1-1
> pve-kernel-4.15: 5.4-12
> pve-kernel-5.3.13-1-pve: 5.3.13-1
> pve-kernel-4.15.18-24-pve: 4.15.18-52
> pve-kernel-4.15.18-21-pve: 4.15.18-48
> pve-kernel-4.15.18-11-pve: 4.15.18-34
> ceph-fuse: 12.2.11+dfsg1-2.1+b1
> corosync: 3.0.2-pve4
> criu: 3.11-3
> glusterfs-client: 5.5-3
> ifupdown: 0.8.35+pve1
> ksm-control-daemon: 1.3-1
> libjs-extjs: 6.0.1-10
> libknet1: 1.13-pve1
> libpve-access-control: 6.0-5
> libpve-apiclient-perl: 3.0-2
> libpve-common-perl: 6.0-9
> libpve-guest-common-perl: 3.0-3
> libpve-http-server-perl: 3.0-3
> libpve-storage-perl: 6.1-3
> libqb0: 1.0.5-1
> libspice-server1: 0.14.2-4~pve6+1
> lvm2: 2.03.02-pve3
> lxc-pve: 3.2.1-1
> lxcfs: 3.0.3-pve60
> novnc-pve: 1.1.0-1
> proxmox-mini-journalreader: 1.1-1
> proxmox-widget-toolkit: 2.1-1
> pve-cluster: 6.1-2
> pve-container: 3.0-15
> pve-docs: 6.1-3
> pve-edk2-firmware: 2.20191127-1
> pve-firewall: 4.0-9
> pve-firmware: 3.0-4
> pve-ha-manager: 3.0-8
> pve-i18n: 2.0-3
> pve-qemu-kvm: 4.1.1-2
> pve-xtermjs: 3.13.2-1
> qemu-server: 6.1-4
> smartmontools: 7.0-pve2
> spiceterm: 3.1-1
> vncterm: 1.6-1
> zfsutils-linux: 0.8.2-pve2
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com<mailto:pve-user at pve.proxmox.com>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpve.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-user&data=02%7C01%7Csmr%40kmi.com%7C5197e0e1f4c64fb738cf08d799a1067b%7Cc2283768b8d34e008f3d85b1b4f03b33%7C0%7C1%7C637146789208877811&sdata=JhYOtpzDpjbQ1g4yrV%2FUwB8d3d4vX08Pd9wISQUmGp8%3D&reserved=0
More information about the pve-user
mailing list