Legacy: ZFS over iSCSI

The ZFS plugin is to be considered as beta

Technology and features

In an upcoming release of proxmox a storage plugin for ZFS will be available providing the ability to use an external storage based on ZFS via iSCSI. The plugin will seamlessly integrate the ZFS storage as a viable storage backend for creating VM's using the the normal VM creation wizard in Proxmox.

When Proxmox creates the raw disk image it will use the plugin to create a ZFS volume as the storage which contains the disk image. Eg. a ZFS volume will be created for every disk image like tank/vm-100-disk-1. Being a native ZFS volume also means that Proxmox will provide users live snapshots and cloning of VM's using ZFS' native snapshot and volume reference features.

Since ZFS is available on several platforms using different iSCSI target implementation the plugin has a number of helper modules each providing the needed iSCSI functionality for the specific platform. For now iSCSI modules exists for the following platforms:

• Solaris based platforms using Comstar. Tested on Omnios and Nexenta Store. For GUI use napp-it or Nexenta.
• BSD based platforms using Istgt. Tested on FreeBSD 8.3, 9.0, 9.1. For GUI use zfsguru.
• Linux based platforms with zfsonlinux using Iet. Tested on Debian Wheezy. I have no knowledge of available GUI's. Edit 2013-10-30: I have begun developing a ZFS plugin for OpenMediaVault in collaboration with the OpenMediaVault team. A beta release of the plugin is scheduled ultimo next month (November 2013).

A word of caution. For enterprise usecases I would only recommend solaris based platforms with Comstar. Linux based platforms can IMHO be used in a non-enterprise setup which requires working HA. I will not recommend BSD based platforms for enterprise and/or HA setups due to limitations in the current iSCSI target implementation. Istgt will require a restart of the daemon every time a LUN is to be deleted or updated which means dropping all current connections. Work has begun to provide a native iSCSI target for FreeBSD 10 which hopefully will solve this inconvenience.

Platform notes

• On all storage notes the following should be added to sshd_config:
  • LookupClientHostnames no
  • VerifyReverseMapping no
  • GSSAPIAuthentication no
• Solaris: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage. Apart from this no other things must be done.
• BSD: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage. Since istgt must have at least one LUN before enabling a target you will have to create one LUN manually. The size is irrelevant so a LUN referencing a volume with size 1MB is sufficient but remember to name the volume with something different than the Proxmox naming scheme to avoid having it show up in the Proxmox content GUI.
• Linux: From all of the Proxmox nodes you must transfer root's ssh key to the storage server using, logged in as root, ssh-copy-id ip_of_storage. Apart from this no other things must be done.
• Nexenta: rm /root/.bash_profile. To avoid to go in nmc console by default.

Proxmox configuration

Gui is not yet available for storage creation.

You need to edit your /etc/pve/storage.cfg

zfs: solaris
	blocksize 4k
	target iqn.2010-08.org.illumos:02:b00c9870-6a97-6f0b-847e-bbfb69d2e581:tank1
	pool tank
	iscsiprovider comstar
	portal 192.168.3.101
	content images

zfs: BSD
	blocksize 4k
	target iqn.2007-09.jp.ne.peach.istgt:tank1
	pool tank
	iscsiprovider istgt
	portal 192.168.3.114
	content images

zfs: linux
	blocksize 4k
	target iqn.2001-04.com.example:tank1
	pool tank
	iscsiprovider iet
	portal 192.168.3.196
	content images

Then you can simply create disk with proxmox gui.

Note: iscsi multipath doesn't work yet, so it's use only the portal ip for the iscsi connection.