[PVE-User] Local XFS storage pool - Fragmentation issues

[Markos Vakondios]

Hello List,

My use case, is to provide a large hardware-RAID backed volume (hence no
ZFS) to PVE for continuous decent performance writes by a running container
(cctv dvr).

I  have 6 x 6TB SATA drives on a hardware RAID 10 configuration.

It 's formated as a ~10TB XFS device:

/dev/sdb on /10TB type xfs (rw,relatime,attr2,inode64,noquota)

I created a local directory storage pool on my PVE host and assigned a
9.8TB mount point  to a running container (DVR)

pve:/10TB/images/101# ls -lah /10TB/images/101/
total 6.3T
drwxr----- 2 root root   30 Feb 15  2017 .
drwxr-xr-x 4 root root   38 Feb 15  2017 ..
-rw-r----- 1 root root 9.8T Oct 18 14:03 vm-101-disk-1.raw

After a couple of months, I started receiving the following XFS errors on
the host:

Oct 17 06:26:09 pve kernel: [5629844.259800] XFS: loop0(28500) possible
memory allocation deadlock size 72832 in kmem_alloc (mode:0x2400240)

leaving me with an unusable filesystem from inside the container (no
writes),  unless I drop the host's page cache:

echo 1 > /proc/sys/vm/drop_caches

or even need to drop all pagecache, dentries and inodes:

echo 3 > /proc/sys/vm/drop_caches (remember to sync first!)

After research, I concluded that the deadlock is caused from heavy
fragmentation on the XFS image file which has 1.2 million extends (!)
according to xfs_bmap.

Of course, there is no way to defrag this filesystem with xfs_fsr, as the
volume file occupies almost 100% of its space (yes, I won't use more than
80%). Also it wouldn't be practical to do so, as the volume is busy 24/7
(cctv recording) and the available capacity matters.

Please correct me if I am wrong, but the above seemed to be a reasonably
decent layout to employ a local XFS filesystem (which is supposed to have
better handling of large files than EXT4) under PVE. But in practice it's
not! Am I missing something?

Is LVM a more suitable solution?

Any ideas or suggestions would be greatly appreciated.

Thanks,

Markos