Legacy: ZFS over iSCSI

The ZFS plugin is Technology Preview in 3.2

Technology and features

In an upcoming release of proxmox a storage plugin for ZFS will be available providing the ability to use an external storage based on ZFS via iSCSI. The plugin will seamlessly integrate the ZFS storage as a viable storage backend for creating VM's using the the normal VM creation wizard in Proxmox.

When Proxmox creates the raw disk image it will use the plugin to create a ZFS volume as the storage which contains the disk image. Eg. a ZFS volume will be created for every disk image like tank/vm-100-disk-1. Being a native ZFS volume also means that Proxmox will provide users live snapshots and cloning of VM's using ZFS' native snapshot and volume reference features.

Since ZFS is available on several platforms using different iSCSI target implementation the plugin has a number of helper modules each providing the needed iSCSI functionality for the specific platform. For now iSCSI modules exists for the following platforms:

• Solaris based platforms using Comstar. Tested on Omnios and Nexenta Store. For GUI use napp-it or Nexenta.
• BSD based platforms using Istgt. Tested on FreeBSD 8.3, 9.0, 9.1. For GUI use zfsguru.
• Linux based platforms with zfsonlinux using Iet. Tested on Debian Wheezy. I have no knowledge of available GUI's. Edit 2013-10-30: I have begun developing a ZFS plugin for OpenMediaVault in collaboration with the OpenMediaVault team. A beta release of the plugin is scheduled ultimo next month (November 2013).

A word of caution. For enterprise usecases I would only recommend solaris based platforms with Comstar. Linux based platforms can IMHO be used in a non-enterprise setup which requires working HA. I will not recommend BSD based platforms for enterprise and/or HA setups due to limitations in the current iSCSI target implementation. Istgt will require a restart of the daemon every time a LUN is to be deleted or updated which means dropping all current connections. Work has begun to provide a native iSCSI target for FreeBSD 10 which hopefully will solve this inconvenience.

Platform notes

• On all zfs storages nodes the following should be added to /etc/ssh/sshd_config:

LookupClientHostnames no 
VerifyReverseMapping no 
GSSAPIAuthentication no

• After libpve-storage-perl-3.0-18 the following procedure must be used. For all storage platforms the distribution of root's ssh key is maintained through Proxmox's cluster wide file system which means you have to create this folder: /etc/pve/priv/zfs. In this folder you place the ssh key to use for each ZFS storage and the name of the key follows this naming scheme: <portal>_id_rsa. Portal is entered in the gui wizard's field portal so if a ZFS storage is referenced via the IP 192.168.1.1 then this IP is entered in the field portal and therefore the key will have this name: 192.168.1.1_id_rsa. Creating the key is simple. As root do the following:

mkdir /etc/pve/priv/zfs
ssh-keygen -f /etc/pve/priv/zfs/192.168.1.1_id_rsa
ssh-copy-id -i /etc/pve/priv/zfs/192.168.1.1_id_rsa root@192.168.1.1

• login once to zfs san from each proxmox node

ssh -i /etc/pve/priv/zfs/192.168.1.1_id_rsa root@192.168.1.1

The authenticity of host '192.168.1.1 (192.168.1.1)' can't be established.
RSA key fingerprint is 8c:f9:46:5e:40:65:b4:91:be:41:a0:25:ef:7f:80:5f.
Are you sure you want to continue connecting (yes/no)? yes

If you are logged in without errors you are ready to use your storage.

• The key creation is only needed once for each portal so if the same portal provides several targets which is used for several storages in Proxmox you only create one key.

• Solaris: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage Still needed in libpve-storage-perl-3.0-18. Apart from this no other things must be done.
• BSD: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage Still needed in libpve-storage-perl-3.0-18. Since istgt must have at least one LUN before enabling a target you will have to create one LUN manually. The size is irrelevant so a LUN referencing a volume with size 1MB is sufficient but remember to name the volume with something different than the Proxmox naming scheme to avoid having it show up in the Proxmox content GUI.
• Linux: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage Still needed in libpve-storage-perl-3.0-18. Apart from this no other things must be done.
• Nexenta: rm /root/.bash_profile. To avoid to go in nmc console by default.

Proxmox configuration

Use the GUI (Datacenter/Storage: Add ZFS) so just edit your /etc/pve/storage.cfg

zfs: solaris
	blocksize 4k
	target iqn.2010-08.org.illumos:02:b00c9870-6a97-6f0b-847e-bbfb69d2e581:tank1
	pool tank
	iscsiprovider comstar
	portal 192.168.3.101
	content images

zfs: BSD
	blocksize 4k
	target iqn.2007-09.jp.ne.peach.istgt:tank1
	pool tank
	iscsiprovider istgt
	portal 192.168.3.114
	content images

zfs: linux
	blocksize 4k
	target iqn.2001-04.com.example:tank1
	pool tank
	iscsiprovider iet
	portal 192.168.3.196
	content images

Then you can simply create disk with proxmox gui.

Note: iscsi multipath doesn't work yet, so it's use only the portal ip for the iscsi connection.