Difference between revisions of "Storage: ZFS"

From Proxmox VE
Jump to navigation Jump to search
Line 299: Line 299:
   zfs clone images2/109@master images2/110
   zfs clone images2/109@master images2/110
== zfs mounting workaround ==
=== zfs mounting workaround ===
2014-01-22  the info below came from this excellent wiki page: <pre> http://wiki.complete.org/ConvertingToZFS </pre>
2014-01-22  the info below came from this excellent wiki page: <pre> http://wiki.complete.org/ConvertingToZFS </pre>

Revision as of 00:51, 23 January 2014


tbd. (running zfs on Proxmox VE is not officially supported)

Native ZFS for Linux on Proxmox

2013-03-27: 0.6.1 ZFSOnLinux (ZoL) is now ready for wide scale deployment on everything from desktops to super computers. See announcement

check  http://zfsonlinux.org/ See the Documentation and Community Resources for more information and help.

using Debian Wheezy packages from zfsonlinux

For PVE on Wheezy [ 3.0+ ].

2013-05-29 - Debian Packages DKMS style packages for Debian are available from the zfsonlinux.org repository. These packages track the latest official upstream tag and are refreshed as new releases are made available from http://zfsonlinux.org/debian.html .

also check http://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/

  • If you are upgrading from the Ubuntu PPA method remove ubuntu packages first.
aptitude remove ubuntu-zfs
  • If this is a new install or you are repeating a previously failed attempt
dpkg --purge zfsonlinux zfsutils libzfs1 zfs-dkms dkms
move the old zfs.list from /etc/apt/sources.list.d
  • make sure pve headers are installed. if not :
aptitude install pve-headers-$(uname -r)
  • I think the ln is still needed.
ln -s /lib/modules/$(uname -r)/build /lib/modules/$(uname -r)/source
then follow instructions from http://zfsonlinux.org/debian.html , i copied here , but check link in case these changed:
su -
wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_1%7Ewheezy_all.deb
dpkg -i zfsonlinux_1~wheezy_all.deb
apt-get update
apt-get install debian-zfs

you should see zfs and spl module builds... check if OK

# ls -l /lib/modules/$(uname -r)/updates/dkms/
total 3128
-rw-r--r-- 1 root root  340944 May 29 10:25 splat.ko
-rw-r--r-- 1 root root  302104 May 29 10:25 spl.ko
-rw-r--r-- 1 root root   13392 May 29 10:27 zavl.ko
-rw-r--r-- 1 root root   71232 May 29 10:27 zcommon.ko
-rw-r--r-- 1 root root 1935120 May 29 10:27 zfs.ko
-rw-r--r-- 1 root root  130408 May 29 10:27 znvpair.ko
-rw-r--r-- 1 root root   40424 May 29 10:27 zpios.ko
-rw-r--r-- 1 root root  330368 May 29 10:27 zunicode.ko


test the above again and update the instructions as necessary.
does dkms auto build modules when pve kernel is upgraded? for that to happen headers would need to be installed and the ln may be needed.

using Ubuntu PPA with Debian Squeeze

Note (2013-03-17): this procedure worked well for PVE 2.3

# apt key
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys  F6B0FC61

echo "deb http://ppa.launchpad.net/zfs-native/stable/ubuntu lucid   main" > /etc/apt/sources.list.d/zfs.list

aptitude update

aptitude install dkms pve-headers-$(uname -r)
  • this is needed for dkms to work:
ln -s /lib/modules/2.6.32-10-pve/build /lib/modules/2.6.32-10-pve/source

Maybe someone who knows how to write a patch could submit one and suggest this be done when kernel is upgraded? Otherwise I think when the Proxmox kernel upgrades, dkms build may fail if this is not done [ using new pve-headers ).

then install zfs

aptitude install ubuntu-zfs

finally make sure that zfs is started at system boot, edit /etc/default/zfs

# ZoL userland configuration.

# Run `zfs mount -a` during system start?
# This should be 'no' if zfs-mountall or a systemd generator is available.

kernel upgrade

currently this needs to be done when a kernel upgrade occurs.

  • before reboot: upgrade from did before reboot [ for you would need to change 2.6.32-11-pve if different version.
aptitude install pve-headers-2.6.32-11-pve
ln -s /lib/modules/2.6.32-11-pve/build /lib/modules/2.6.32-11-pve/source
  • this will force modules to be built ( there is probably a better way to do this , so correct this if you know how to make dkms do the build ).
aptitude reinstall spl-dkms  zfs-dkms

using git

  • install these
aptitude install build-essential gawk alien fakeroot zlib1g-dev uuid uuid-dev libssl-dev parted  pve-headers-$(uname -r) 
  • get make and install spl
cd /usr/src
git clone https://github.com/zfsonlinux/spl.git
cd spl
make deb
dpkg -i *.deb
modprobe spl
  • get make and install zfs
cd /usr/src
git clone https://github.com/zfsonlinux/zfs.git
cd zfs
make deb
dpkg -i *.deb
  • make sure module can load
modprobe zfs
  • install debs does not do this:
update-rc.d zfs defaults
  • note , the deb files made above can be used to install zfs on another system.
  • if there is a pre existing zfs pool then simply use the next line to recover it. this can happen if you re-install proxmox and have zfs disks from before.
zpoo  </cman>

  • If there is a need or want I could set up a repository with pre build debs. Just send an email .

Creating Pools and Filesystems

use raidz2 or mirror in main data systems.

Create a raidz pool named “tank”

zpool create  -f -o ashift=12 tank  raidz  /dev/sda  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf  /dev/sdg  /dev/sdh
  • note this happened to 2 or 3 disks the 1-st time. these were disks from a 3-ware raid.
zpool create  -f -o ashift=12 tank  raidz2   /dev/sda  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf  /dev/sdg  /dev/sdh
invalid vdev specification
the following errors must be manually repaired:
cannot stat /dev/sdg2: No such file or directory

had to use fdisk to add partitions 1 , 2 etc to those disks . I forget if I left the partitions there or not. but had to add to get this fixed.

NOTE: This is quite risky if you intend to deploy ZFS on a server. Any changes to the operating system's devices layout will break your pool. Don't create a pool using device names like /dev/sda. For persistence, use devices under /dev/disk/by-id. - Bryan Baldwin

Create directories and mount points

zfs create tank/data
zfs set mountpoint=/data tank/data
  • mount all
zfs mount -a


to start over

zpool destroy zfs-pool

Setting Properties

  • Automatically NFS-export all home directories [ I do not know how to use zfs nfs yet , or if it works yet in linux...].
zfs set sharenfs=rw tank/home
  • check sharenfs
zfs get sharenfs tank/home
tank/home  sharenfs  rw        local
  • Turn on compression for everything in the pool
zfs set compression=on tank

kvm tuning

see thread on prox forum , per user Nemesiz :

  • pool:
zfs set primarycache=all tank
  • kvm config:
  • change cache to Write Back
You can do it using web GUI or manually. Example:
ide0: data_zfs:100/vm-100-disk-1.raw,cache=writeback

if not set this happened:

qm start 4016
kvm: -drive file=/data/pve-storage/images/4016/vm-4016-disk-1.raw,if=none,id=drive-virtio1,aio=native,cache=none: could not open disk image /data/pve-storage/images/4016/vm-4016-disk-1.raw: Invalid argument

Install on a high performance system

As of 2013 high performance servers have 16-64 cores, 256GB-1TB RAM and potentially many 2.5" disks and/or a PCIe based SSD with half a million IOPS. High performance systems benefit from a number of custom settings, for example enabling compression typically improves performance.

  • If you have a good number of disks keep organized by using aliases. Edit /etc/zfs/vdev_id.conf to prepare aliases for disk devices found in /dev/disk/by-id/ :
# run 'udevadm trigger' after updating this file
alias a0        scsi-36848f690e856b10018cdf39854055206
alias b0        scsi-36848f690e856b10018cdf3ce573fdeb6
alias a1        scsi-36848f690e856b10018cdf40f5b277cbc
alias b1        scsi-36848f690e856b10018cdf43a5db1b99b
alias a2        scsi-36848f690e856b10018cdf4575f652ad0
alias b2        scsi-36848f690e856b10018cdf47761587cec

Use flash for caching/logs. If you have only one SSD, use cfdisk to create a small partion for the ZIL (ZFS intent log) and a larger one for the L2ARC (ZFS read cache on disk). Make sure that the ZIL is on the first partition. In our case we have a Express Flash PCIe SSD with 175GB capacity and setup a ZIL with 25GB and a L2ARC cache partition of 150GB

  • edit /etc/modprobe.d/zfs.conf​ to apply several tuning options for high performance servers:
# ZFS tuning for a proxmox machine that reserves 64GB for ZFS
# Don't let ZFS use less than 4GB and more than 64GB
options zfs zfs_arc_min=4294967296
options zfs zfs_arc_max=68719476736
# disabling prefetch is no longer required
options zfs l2arc_noprefetch=0
  • create a zpool of striped mirrors (equivalent to RAID10) with log device and cache and always enable compression:
zpool create -o compression=on -f tank mirror a0 b0 mirror a1 b1 mirror a2 b2 log /dev/rssda1 cache /dev/rssda2​
  • ​​​​​​​​check the status of the newly created pool:
root@proxmox:/# zpool status
  pool: tank
 state: ONLINE
  scan: none requested

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            a0      ONLINE       0     0     0
            b0      ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            a1      ONLINE       0     0     0
            b1      ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            a2      ONLINE       0     0     0
            b2      ONLINE       0     0     0
          rssda1    ONLINE       0     0     0
          rssda2    ONLINE       0     0     0

errors: No known data errors

Using PVE 2.3 on a 2013 high performance system with ZFS you can install Windows Server 2012 Datacenter Edition with GUI in just under 4 minutes.

zfs links and docs

and this has some very important information to know before implementing zfs on a production system.

check these well written manual pages:

man zfs

man zpool

How to install ZFS-fuse under Proxmox

(Tested with Kernel 2.6.32)

 apt-get install build-essential libaio-dev libattr1-dev libacl1-dev libz-dev libz-dev libfuse-dev libfuse2 scons libssl-dev
 wget -nd http://zfs-fuse.net/releases/0.6.0/zfs-fuse-0.6.0.tar.bz2
 tar jxvf zfs-fuse-0.6.0.tar.bz2
 cd zfs-fuse-0.6.0
 cd src
 scond install

Note: zfs-fuse-0.6.0 is now (Dec 2012) deprecated. Use zfs-fuse-0.7.0.tar.bz2 instead.

Fire up ZFS-fuse daemon


Create Zpool

 zpool create -m /var/lib/vz/images2 /dev/sdb

Create ZFS

 zfs create images2/109

ZFS clone a vm 109 to vm 110

 zfs snapshot images2/109@master
 zfs clone images2/109@master images2/110

zfs mounting workaround

2014-01-22 the info below came from this excellent wiki page:


this is just a quick copy paste of notes i used on 3 systems. I'm glad that a Debian Developer is using zfs and fixing these kinds of things.

see http://wiki.complete.org/ConvertingToZFS?action=newaccount

Edit /etc/default/zfs and set ZFS_MOUNT='yes'

edit /etc/insserv.conf, 
 and at the end of the $local_fs line, 
 add zfs-mount (without a plus).

edit /etc/init.d/zfs-mount and find three lines near the top, changing them like this:
# Required-Start:
# Required-Stop:
# Default-Start: S

Activating init.d changes
Then run:
insserv -v -d zfs-mount

I had an issue with pve storage on zfs, before pve would start before zfs and create directories at the zfs mount point. to fix that start single user mode and remove the directories [ make sure they are empty.... ].