ZFS: Tips and Tricks: Difference between revisions

From Proxmox VE
Jump to navigation Jump to search
m (Reverted edits by Emmanuel Kasper (talk) to last revision by Dietmar)
(remove outdated section about disk replacement in root pool and link to new section in admin guide)
 
(12 intermediate revisions by 7 users not shown)
Line 1: Line 1:
==See Also==
[[ZFS on Linux]]
[[Storage: ZFS]]
== Changing a failed Device ==
zpool replace -f <pool-name> <old device> <new-device>
== Using ZFS Storage Plugin (via Proxmox VE GUI or shell) ==
== Using ZFS Storage Plugin (via Proxmox VE GUI or shell) ==
If the zpool is created, you can use it on Proxmox VE GUI and CLI.
After the ZFS pool has been created, you can add it with the Proxmox VE GUI or CLI.
=== Adding a ZFS storage via GUI ===
 
Go to Datacenter/Storage and use the ZFSStorage plugin to add your zpool (select ZFS).
 
*ID is for identification of the Storage
*the checkbox ZFS Pool shows all existing pools (use CLI to create more)
*Thin provisioning: allocate not all space immediately by creating virtual disks


=== Adding a ZFS storage via CLI ===
=== Adding a ZFS storage via CLI ===
To create it by CLI use
To create it by CLI use:
  pvesm add zfspool <storage-ID> -pool <pool-name>
  pvesm add zfspool <storage-ID> -pool <pool-name>


=== Adding ZFS root file-system as storage with Plugin  ===
=== Adding a ZFS storage via Gui ===
If you install Proxmox with ZFS, VM disks on local storage will be simple files, so you will have 2 layers of file system though reducing performance. This is especially true if you choose qcow2 format, because then you will have a "copy on write" image disk (qcow2) that writes on a "copy on write" file system (ZFS).
To add it with the GUI:
To avoid it, thanks to the flexibility of ZFS, you can create an additional file system on the default storage pool and, thanks to the ZFS Plugin, use it as "block device" storage have VMs disks created as volumes inside ZFS. In such configuration you keep the ZFS benefit of snapshots and checksuming.
Go to the datacenter, add storage, select ZFS.
To obtain is do the following:
Create a new filesystem (it's better have a distinct one, separate from the ROOT one), that we will call "zfsdisks" in this example, from shell:
zfs create rpool/zfsdisks
Now add it to the storage (Datacenter -> [Storage] -> Add, choose "ZFS", ad ID let's call it, for example, "zfsvols", as "ZFS Pool" choose "rpool/zfsdisks", set "thin provisioning" and you are ok.
When you create a VM choose "zfsvols" as storage.
cat /etc/pve/storage.cfg
zfspool: zfsvols
        pool rpool/zfsdisks
        content images,rootdir
        sparse
 
If you want to add the storage from the shell, you will have the same result (even if /etc/pve/storage.cfg does not exists yet) with
pvesm add zfspool zfsvols -pool rpool/zfsdisks -content images,rootdir -sparse
With a VM 100 and one disk you will have something like that
# zfs list
NAME                          USED  AVAIL  REFER  MOUNTPOINT
rpool                        4.78G  26.0G    96K  /rpool
rpool/ROOT                    676M  26.0G    96K  /rpool/ROOT
rpool/ROOT/pve-1              676M  26.0G  676M  /
rpool/swap                    4.12G  30.1G    64K  -
rpool/zfsdisks                160K  26.0G    96K  /rpool/zfsdisks
rpool/zfsdisks/vm-100-disk-1    64K  26.0G    64K  -
so you see that it's a "block device" and thin provisioning is used (see 'USED' column that states 64K)


== Misc ==
== Misc ==
=== QEMU tuning ===
=== QEMU disk cache mode ===
see thread on proxmox forum, per user Nemesiz:
If you get the warning:
*pool:
zfs set primarycache=all tank
*kvm config:
* change cache to Write Back
:You can do it using web GUI or manually. Example:
ide0: data_zfs:100/vm-100-disk-1.raw,cache=writeback
if not set this happened:
<pre>
<pre>
qm start 4016
qm start 4016
kvm: -drive file=/data/pve-storage/images/4016/vm-4016-disk-1.raw,if=none,id=drive-virtio1,aio=native,cache=none: could not open disk image /data/pve-storage/images/4016/vm-4016-disk-1.raw: Invalid argument
kvm: -drive file=/data/pve-storage/images/4016/vm-4016-disk-1.raw,if=none,id=drive-virtio1,aio=native,cache=none: could not open disk image /data/pve-storage/images/4016/vm-4016-disk-1.raw: Invalid argument
</pre>
</pre>
(On PVE 4.x, you may also see warnings about the filesystem not supporting O_DIRECT.)
or a warning that the filesystem do not supporting O_DIRECT, set the disk cache type of your VM from '''none''' to '''writeback'''.
 
=== LXC with ACL on ZFS ===
ZFS uses as default store for ACL hidden files on filesystem.
This reduces performance enormously and with several thousand files a system can feel unresponsive.
Storing the xattr in the inode will revoke this performance issue.
 
Modification to do
zfs set xattr=sa dnodesize=auto vmstore/data
 
'''Warning:''' Do not set dnodesize on rpool because GRUB is not able to handle a different size.
see Bug entry https://savannah.gnu.org/bugs/?func=detailitem&item_id=48885


=== Example configurations for running Proxmox VE with ZFS ===
=== Example configurations for running Proxmox VE with ZFS ===
Line 164: Line 128:
Now set /mnt/vztmp in your /etc/vzdump.conf for tmp
Now set /mnt/vztmp in your /etc/vzdump.conf for tmp


=== Replacing a failed disk in the root pool ===
[[ZFS_on_Linux#sysadmin_zfs_change_failed_dev|This is explained in the ZFS on Linux chapter of the Administration Guide.]]
== Glossary ==
== Glossary ==
*ZPool is the logical unit of the underlying disks, what zfs use.
*ZPool is the logical unit of the underlying disks, what zfs use.
Line 185: Line 151:
  man zpool
  man zpool


[[Category:HOWTO]] [[Category:Technology]]
== See also ==
 
[[ZFS on Linux]]
 
[[Storage: ZFS]]
 
[[Category:HOWTO]]

Latest revision as of 13:09, 28 July 2020

Using ZFS Storage Plugin (via Proxmox VE GUI or shell)

After the ZFS pool has been created, you can add it with the Proxmox VE GUI or CLI.

Adding a ZFS storage via CLI

To create it by CLI use: 

pvesm add zfspool <storage-ID> -pool <pool-name>

Adding a ZFS storage via Gui

To add it with the GUI: Go to the datacenter, add storage, select ZFS.

Misc

QEMU disk cache mode

If you get the warning: 

qm start 4016
kvm: -drive file=/data/pve-storage/images/4016/vm-4016-disk-1.raw,if=none,id=drive-virtio1,aio=native,cache=none: could not open disk image /data/pve-storage/images/4016/vm-4016-disk-1.raw: Invalid argument

or a warning that the filesystem do not supporting O_DIRECT, set the disk cache type of your VM from none to writeback.

LXC with ACL on ZFS

ZFS uses as default store for ACL hidden files on filesystem. This reduces performance enormously and with several thousand files a system can feel unresponsive. Storing the xattr in the inode will revoke this performance issue.

Modification to do 

zfs set xattr=sa dnodesize=auto vmstore/data

Warning: Do not set dnodesize on rpool because GRUB is not able to handle a different size. see Bug entry https://savannah.gnu.org/bugs/?func=detailitem&item_id=48885

Example configurations for running Proxmox VE with ZFS

Install on a high performance system

As of 2013 and later, high performance servers have 16-64 cores, 256GB-1TB RAM and potentially many 2.5" disks and/or a PCIe based SSD with half a million IOPS. High performance systems benefit from a number of custom settings, for example enabling compression typically improves performance.

  • If you have a good number of disks keep organized by using aliases. Edit /etc/zfs/vdev_id.conf to prepare aliases for disk devices found in /dev/disk/by-id/ :
# run 'udevadm trigger' after updating this file
alias a0        scsi-36848f690e856b10018cdf39854055206
alias b0        scsi-36848f690e856b10018cdf3ce573fdeb6
alias a1        scsi-36848f690e856b10018cdf40f5b277cbc
alias b1        scsi-36848f690e856b10018cdf43a5db1b99b
alias a2        scsi-36848f690e856b10018cdf4575f652ad0
alias b2        scsi-36848f690e856b10018cdf47761587cec

Use flash for caching/logs. If you have only one SSD, use parted of gdisk to create a small partition for the ZIL (ZFS intent log) and a larger one for the L2ARC (ZFS read cache on disk). Make sure that the ZIL is on the first partition. In our case we have a Express Flash PCIe SSD with 175GB capacity and setup a ZIL with 25GB and a L2ARC cache partition of 150GB.

  • edit /etc/modprobe.d/zfs.conf to apply several tuning options for high performance servers:
# ZFS tuning for a proxmox machine that reserves 64GB for ZFS
#
# Don't let ZFS use less than 4GB and more than 64GB
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=68719476736
#
# disabling prefetch is no longer required
options zfs l2arc_noprefetch=0
  • create a zpool of striped mirrors (equivalent to RAID10) with log device and cache and always enable compression:
zpool create -o compression=on -f tank mirror a0 b0 mirror a1 b1 mirror a2 b2 log /dev/rssda1 cache /dev/rssda2
  • check the status of the newly created pool:
root@proxmox:/# zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            a0      ONLINE       0     0     0
            b0      ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            a1      ONLINE       0     0     0
            b1      ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            a2      ONLINE       0     0     0
            b2      ONLINE       0     0     0
        logs
          rssda1    ONLINE       0     0     0
        cache
          rssda2    ONLINE       0     0     0

errors: No known data errors

Using PVE 2.3 on a 2013 high performance system with ZFS you can install Windows Server 2012 Datacenter Edition with GUI in just under 4 minutes.


Troubleshooting and known issues

ZFS packages are not installed

If you upgraded to 3.4 or later, zfsutils package is not installed. You can install it with apt: 

apt-get install zfsutils zfs-initramfs

Grub boot ZFS problem

  • Symptoms: stuck at boot with an blinking prompt.
  • Reason: If you ZFS raid it could happen that your mainboard does not initial all your disks correctly and Grub will wait for all RAID disk members - and fails. It can happen with more than 2 disks in ZFS RAID configuration - we saw this on some boards with ZFS RAID-0/RAID-10

Boot fails and goes into busybox

If booting fails with something like 

No pool imported. Manually import the root pool
at the command prompt and then exit.
Hint: try: zpool import -R /rpool -N rpool

is because zfs is invoked too soon (it has happen sometime when connecting a SSD for future ZIL configuration). To prevent it there have been some suggestions in the forum. Try to boot following the suggestions of busybox or searching the forum, and try ONE of the following:

a) edit /etc/default/grub and add "rootdelay=10" at GRUB_CMDLINE_LINUX_DEFAULT (i.e. GRUB_CMDLINE_LINUX_DEFAULT="rootdelay=10 quiet") and then issue a # update-grub

b) edit /etc/default/zfs, set ZFS_INITRD_PRE_MOUNTROOT_SLEEP='4', and then issue a "update-initramfs -k 4.2.6-1-pve -u"


Snapshot of LXC on ZFS

If you can't create a snapshot of an LXC container on ZFS and you get following message: 

INFO: rsync: set_acl: sys_acl_set_file(archiv, ACL_TYPE_DEFAULT): Operation not supported (95)

you can run following commands 

zfs create -o mountpoint=/mnt/vztmp rpool/vztmp
zfs set acltype=posixacl rpool/vztmp

Now set /mnt/vztmp in your /etc/vzdump.conf for tmp

Replacing a failed disk in the root pool

This is explained in the ZFS on Linux chapter of the Administration Guide.

Glossary

  • ZPool is the logical unit of the underlying disks, what zfs use.
  • ZVol is an emulated Block Device provided by ZFS
  • ZIL is ZFS Intent Log, it is a small block device ZFS uses to write faster
  • ARC is Adaptive Replacement Cache and located in Ram, its the Level 1 cache.
  • L2ARC is Layer2 Adaptive Replacement Cache and should be on an fast device (like SSD).

Further readings about ZFS

and this has some very important information to know before implementing zfs on a production system.

Very well written manual pages 

man zfs
man zpool

See also

ZFS on Linux

Storage: ZFS

Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=ZFS:_Tips_and_Tricks&oldid=10806"