Storage: ZFS: Difference between revisions

From Proxmox VE
Jump to navigation Jump to search
No edit summary
 
(149 intermediate revisions by 16 users not shown)
Line 1: Line 1:
=Native ZFS for Linux on Proxmox 2.0=
<!--PVE_IMPORT_START_MARKER-->
*2012-02-25.  check  http://zfsonlinux.org/ See the Documentation and Community Resources for more information and help.
<!-- Do not edit - this is autogenerated content -->
{{#pvedocs:pve-storage-zfspool-plain.html}}
I had tried 3 other operating systems including Debian/kFreebsd and for each was able to recover all the zfs  file systems from previous installs. Even with disk name changes [bsd uses ad11, linux sdb ] no data was lost. Today I set up zfsonlinux using the following info on 2 prox 2.0 systems
[[Category:Reference Documentation]]
 
<pvehide>
== dkms install ==
Storage pool type: zfspool
<pre>
This backend allows you to access local ZFS pools (or ZFS file systems
# apt key
inside such pools).
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys  F6B0FC61
Configuration
 
The backend supports the common storage properties content, nodes,
#sources:
disable, and the following ZFS specific properties:
echo "deb http://ppa.launchpad.net/zfs-native/stable/ubuntu lucid  main" > /etc/apt/sources.list.d/zfs.list
pool
 
Select the ZFS pool/filesystem. All allocations are done within that
aptitude update
pool.
 
blocksize
# some or all of these are needed
Set ZFS blocksize parameter.
aptitude install dkms
sparse
 
Use ZFS thin-provisioning. A sparse volume is a volume whose
# i had done this before...  dkms install should install what is needed..
reservation is not equal to the volume size.
# aptitude install build-essential gawk alien fakeroot zlib1g-dev uuid uuid-dev libssl-dev parted  pve-headers-$(uname -r)
mountpoint
 
The mount point of the ZFS pool/filesystem. Changing this does not
</pre>
affect the mountpoint property of the dataset seen by zfs.
 
Defaults to /&lt;pool&gt;.
 
Configuration Example (/etc/pve/storage.cfg)
this needs to be done manually.
zfspool: vmdata
Maybe someone who knows how to write a patch could submit one and suggest this be done when kernel is upgraded?
        pool tank/vmdata
<pre>
        content rootdir,images
ln -s /lib/modules/2.6.32-10-pve/build /lib/modules/2.6.32-10-pve/source
        sparse
</pre>
File naming conventions
 
The backend uses the following naming scheme for VM images:
then install zfs
vm-&lt;VMID&gt;-&lt;NAME&gt;      // normal VM images
<pre>
base-&lt;VMID&gt;-&lt;NAME&gt;    // template VM image (read-only)
aptitude install ubuntu-zfs
subvol-&lt;VMID&gt;-&lt;NAME&gt;  // subvolumes (ZFS filesystem for containers)
</pre>
&lt;VMID&gt;
 
This specifies the owner VM.
== using git ==
&lt;NAME&gt;
*install these
This can be an arbitrary name (ascii) without white space. The
aptitude install build-essential gawk alien fakeroot zlib1g-dev uuid uuid-dev libssl-dev parted  pve-headers-$(uname -r)
backend uses disk[N] as default, where [N] is replaced by an
*get make and install spl
integer to make the name unique.
<pre>
Storage Features
cd /usr/src
ZFS is probably the most advanced storage type regarding snapshot and
git clone https://github.com/zfsonlinux/spl.git
cloning. The backend uses ZFS datasets for both VM images (format
cd spl
raw) and container data (format subvol). ZFS properties are
./configure
inherited from the parent dataset, so you can simply set defaults
make deb
on the parent dataset.
dpkg -i *.deb
Table 1. Storage features for backend zfs
modprobe spl
Content types
</pre>
Image formats
*get make and install zfs
Shared
<pre>
Snapshots
cd /usr/src
Clones
git clone https://github.com/zfsonlinux/zfs.git
images rootdir
cd zfs
raw subvol
./configure
no
make deb
yes
dpkg -i *.deb
yes
</pre>
Examples
 
It is recommended to create an extra ZFS file system to store your VM images:
* make sure module can load
# zfs create tank/vmdata
modprobe zfs
To enable compression on that newly allocated file system:
 
# zfs set compression=on tank/vmdata
* install debs does not do this:
You can get a list of available ZFS filesystems with:
update-rc.d zfs defaults
# pvesm zfsscan
 
See Also
*'''note , the deb files made above can be used to install zfs on another system'''.
Storage
 
ZFS on Linux
*if there is a pre existing zfs pool then simply use the next line to recover it. this can happen if you re-install proxmox and have zfs disks from before.
</pvehide>
zpoo  </cman>
<!--PVE_IMPORT_END_MARKER-->
 
 
*If there is a need or want I could set up a repository with pre build debs. Just send an email .
 
=Creating Pools and Filesystems=
'''use raidz2 or mirror in main data systems.'''
 
==Create a raidz pool named “tank”==
zpool create  -f -o ashift=12 tank  raidz  /dev/sda  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf  /dev/sdg  /dev/sdh
 
*note this happened to 2 or 3 disks the 1-st time. these were disks from a 3-ware raid.
<pre>
zpool create  -f -o ashift=12 tank  raidz2  /dev/sda  /dev/sdb  /dev/sdc  /dev/sdd  /dev/sde  /dev/sdf  /dev/sdg  /dev/sdh
invalid vdev specification
the following errors must be manually repaired:
cannot stat /dev/sdg2: No such file or directory
</pre>
had to use fdisk to add partitions 1 , 2 etc to those disks .   I forget if I left the partitions there or not. but had to add to get this fixed.
 
==Create directories and mount points==
<pre>
zfs create tank/data
zfs set mountpoint=/data tank/data
</pre>
 
*mount all
zfs mount -a
 
==destroy==
to start over
zpool destroy zfs-pool
 
==Setting Properties==
*Automatically NFS-export all home directories  [ I do not know how to use zfs nfs yet , or if it works yet in linux...].
zfs set sharenfs=rw tank/home
*check sharenfs 
<pre>
zfs get sharenfs tank/home
NAME    PROPERTY  VALUE    SOURCE
tank/home  sharenfs  rw        local
</pre>
*Turn on compression for everything in the pool
zfs set compression=on tank
 
== kvm tuning==
see thread on prox forum , per user Nemesiz  :
*pool:
zfs set primarycache=all tank
*kvm config:
* change cache to Write Back
:You can do it using web GUI or manually. Example:
ide0: data_zfs:100/vm-100-disk-1.raw,cache=writeback
if not set this happened:
<pre>
qm start 4016
kvm: -drive file=/data/pve-storage/images/4016/vm-4016-disk-1.raw,if=none,id=drive-virtio1,aio=native,cache=none: could not open disk image /data/pve-storage/images/4016/vm-4016-disk-1.raw: Invalid argument
</pre>
 
=zfs links and docs=
*http://zfsonlinux.org/faq.html
 
*http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/zfslast.pdf
 
and this has some very important information to know before implementing zfs on a production  system.
*http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
 
check these well written manual pages:
man zfs
man zpool
 
= How to install ZFS-fuse under Proxmox =
 
(Tested with Kernel 2.6.32)
 
  apt-get install build-essential libaio-dev libattr1-dev libacl1-dev libz-dev libz-dev libfuse-dev libfuse2 scons libssl-dev
  wget -nd http://zfs-fuse.net/releases/0.6.0/zfs-fuse-0.6.0.tar.bz2
  tar jxvf zfs-fuse-0.6.0.tar.bz2
  cd zfs-fuse-0.6.0
  cd src
  scons
  scond install
 
Fire up ZFS-fuse daemon
  /usr/local/sbin/zfs-fuse
 
Create Zpool
  zpool create -m /var/lib/vz/images2 /dev/sdb
 
Create ZFS
  zfs create images2/109
 
ZFS clone a vm 109 to vm 110
  zfs snapshot images2/109@master
  zfs clone images2/109@master images2/110
 
= Debian GNU/kFreeBSD =
'''As of 2012-02 the squeeze version does not have NFS, so AFAIK can not be used for Proxmox storage.''' When  Debsd has nfs support then I'd use this for storage of iso's and backups to start.  In the mean time this works as a samba server..
 
zfs works much better using  Debian/GNU kFreeBSD  then fuse.
 
after installing the system to kvm or hardware:
 
*add some disks .  in KVM i was only able to get ide to work.
 
*the scsi disks look like:
<pre>
ls /dev/ad*
/dev/ad10  /dev/ad10s1  /dev/ad10s2  /dev/ad11  /dev/ad12  /dev/ad13
</pre>
 
*  per http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide  - if I read it right - use entire disk not slices for zfs (I assume slices = partitions).
 
*if the disks have partition tables, then clear them. just not the operating system disk!   I used fdisk to clear the tables. This may not be necessary , if it is not then please delete this part.
<pre>
fdisk /dev/ad11
o
w
</pre>
then same for ad12 and ad13
 
*install this
aptitude install zfsutils
 
*make a pool
zpool create -f  zfs-pool /dev/ad11 /dev/ad12  /dev/ad13
 
df | grep zfs-pool
zfs-pool      zfs    1.4T  18K  1.4T  1% /zfs-pool
 
*http://wiki.debian.org/Debian_GNU/kFreeBSD

Latest revision as of 08:08, 6 December 2019

Storage pool type: zfspool

This backend allows you to access local ZFS pools (or ZFS file systems inside such pools).

Configuration

The backend supports the common storage properties content, nodes, disable, and the following ZFS specific properties:

pool

Select the ZFS pool/filesystem. All allocations are done within that pool.

blocksize

Set ZFS blocksize parameter.

sparse

Use ZFS thin-provisioning. A sparse volume is a volume whose reservation is not equal to the volume size.

mountpoint

The mount point of the ZFS pool/filesystem. Changing this does not affect the mountpoint property of the dataset seen by zfs. Defaults to /<pool>.

Configuration Example (/etc/pve/storage.cfg)
zfspool: vmdata
        pool tank/vmdata
        content rootdir,images
        sparse

File naming conventions

The backend uses the following naming scheme for VM images:

vm-<VMID>-<NAME>      // normal VM images
base-<VMID>-<NAME>    // template VM image (read-only)
subvol-<VMID>-<NAME>  // subvolumes (ZFS filesystem for containers)
<VMID>

This specifies the owner VM.

<NAME>

This can be an arbitrary name (ascii) without white space. The backend uses disk[N] as default, where [N] is replaced by an integer to make the name unique.

Storage Features

ZFS is probably the most advanced storage type regarding snapshot and cloning. The backend uses ZFS datasets for both VM images (format raw) and container data (format subvol). ZFS properties are inherited from the parent dataset, so you can simply set defaults on the parent dataset.

Table 1. Storage features for backend zfs
Content types Image formats Shared Snapshots Clones

images rootdir

raw subvol

no

yes

yes

Examples

It is recommended to create an extra ZFS file system to store your VM images:

# zfs create tank/vmdata

To enable compression on that newly allocated file system:

# zfs set compression=on tank/vmdata

You can get a list of available ZFS filesystems with:

# pvesm zfsscan

See Also


Retrieved from "https://pve.proxmox.com/mediawiki/index.php?title=Storage:_ZFS&oldid=10577"