With the Proxmox VE ZFS replication manager (pve-zsync) you can synchronize your virtual machine (virtual disks and VM configuration) or directory stored on ZFS between two servers. By synchronizing, you have a full copy of your virtual machine on the second host and you can start your virtual machines on the second server (in case of data loss on the first server).
By default, the tool syncs every 15 minutes, but the synchronization interval is fully configurable via the integrated cron job setup. The configuration of pve-zsync can be done either on the source server or on the target server.
This is useful for advanced backup strategies.
Note: pve-zsync was introduced in Proxmox VE 3.4 as technology preview. The package can also be installed on plain Debian Wheezy, Jessie or Stretch servers, as long as ZFS is configured.
Note: Our ZFS is configured to auto-mount all subvols, so keep this in mind if you use pve-zsync. Also, zvol will be scanned by lvm.
- Speed limiter
- Syncing interval can be set by cron
- Syncing VM (disks and config) but also ZFS Datasets
- Can keep multiple backups
- Can be used in both directions
- Can send on local host
- Traffic is encrypted
- not possible to sync recursive
- only ssh for transfer
- email notification is done by cron
- Name resolution not taken into account, you have to use IP Addresses, even using hosts file hostnames will not work
- Both, target and source server must support ZFS (best practice: use Proxmox VE hosts).
- SSH must be installed and configured
- to receive email-notifications, a working mail server is required (e.g. postfix).
- JASON.pm (libjson-perl this is the Debian package name, e.g. needed on Wheezy)
PVE Storage Replication and PVE-zsync
PVE Storage Replication and PVE-zsync work completely independent and do not harm each other, as long the following requirements are followed:
- The destination pool / subset are different.
- You do not migrate a guest to an other node.
Sum up of the differences:
|Characteristic||PVE Storage Replication||PVE-zsync|
|Replication||Cluster Wide||to every Node which meets the requirements|
|Operation mode||push||push or pull|
|Management||GUI and Command line||Command line only|
|Main Goal||redundancy||offsite backup|
Migrate form pve-zsync to Storage Replication
When you have a cluster and would like to switch to Storage Replication, you have to destroy the pve-zsync job, clean up the storage and create a new Storage Replication job.
pve-zsync destroy <vmid> [--jobname <test>]
Then you have to cleanup the zfs storage.
Start at the source side and remove all snapshots which start with '@rep_'
zfs destroy <pool>/[<path>/]vm-<VMID>-<type>-<number>@<rep_snapshots>
Then destroy all guest datasets on the destination side.
zfs destroy -R vm-<vmid>-*-<DiskNO>
Now you can create a new Storage Replication job More information about see Storage Replication
Configuration and use
Install the package with apt, on your Proxmox VE host:
apt-get install pve-zsync
This tool need basically no configuration. On the the first usage, when you create a job with a unknown host, the tool will ask you about the password for the remote server.
Sync a VM or ZFS dataset one time
(N.B. this is also possible if a recurring job for that VM already exists, here you must have in mind that the naming in --name must be the same).
root@zfs1:~# pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 2 --name test1 --limit 512
This command sync VM 100 witch is located on the server where the tool is called and sent it to the server 192.168.1.2 on the zpool tank which has an subset backup. The --maxsnap tells that it should be kept 2 backups, if there are more then 2 backups, the 3 one will be erased (sorted by creation time). Name is only needed if there is already a sync job. The --limit param sets the speed limit what is used for syncing, here it would be 512 KBytes/s.
Create a recurring sync job
root@zfs2:~# pve-zsync create --source 192.168.1.1:100 --dest tank/backup --verbose --maxsnap 2 --name test1 --limit 512 --skip
The --skip parameter disables the initial sync, that normally would be done immediately but can take a while, depending on the size of the backup. The initial sync will be done at the first sync time.
Delete a recurring sync job
if you delete a job, the former backup data will not be erased only the config will be erased.
root@zfs2:~# pve-zsync destroy --source 192.168.1.1:100 --name test1
Name is not necessary if it is default.
Pause a sync job
If you want to pause a job, say maintenance the source server!
root@zfs2:~# pve-zsync disable --source 192.168.1.1:100 --name test1
Reactivate a sync job
to reactivate a job, because the job was paused or the job failed use.
root@zfs2:~# pve-zsync enable --source 192.168.1.1:100 --name test1
This will reset the error flag in case of failure.
You can edit the configuration in /etc/cron.d/pve-zsync or destroy the job and create it new.
Information about the jobs
To get a overview about all jobs use:
root@zfs94:~# pve-zsync list SOURCE NAME STATE LAST SYNC TYPE 192.168.15.95:100 testing1 ok 2015-05-13_14:44:00 ssh 192.168.15.95:data/test1 testing1 syncing2015-05-13_14:44:11 ssh root@zfs94:~# pve-zsync status SOURCE NAME STATUS 192.168.15.95:100 testing1 ok 192.168.15.95:data/test1 testing1 syncing
Recovering an VM
You must recover the VM or dataset manually. (In one of the upcoming releases, the restore for Proxmox VE VM will be integrated).
Fist, stop the sync job for the VM or dataset in question.
NOTE: if not you can be interfere the sync job or your snapshot will removed before you are able to send it.
root@zfs2:~# pve-zsync disable --source 192.168.15.1:100 --name test root@zfs2:~# pve-zsync list SOURCE NAME STATE LAST SYNC TYPE 192.168.15.1:100 test stopped2015-06-12_11:03:01 ssh
Then you can send the VM or Dataset to the selected target. SSH is only needed if you send to a remote sever.
zfs send <pool>/[<path>/]vm-<VMID>-disk-<number>@<last_snapshot> | [ssh root@<destination>] zfs receive <pool>/<path>/vm-<VMID>-disk-<number>
If you have a VM you must also copy the config and you need to correct the virtual disk storage configuration accordingly.
cp /var/lib/pve-zsync/<VMID>.conf.rep_<JOB_NAME><VMID>_<TIMESTAMP> /etc/pve/qemu-server/<VMID>.conf
NOTE: On PVE 4.2+ the path is: /var/lib/pve-zsync/data/<VMID>.conf.rep_<JOB_NAME><VMID>_<TIMESTAMP>
Example restore VM 100 with 2 disk from 192.168.15.2 (pve2) to 192.168.15.1 (pve1) and change VMID from 100 to 200:
root@zfs2:~# zfs send rpool/backup/vm-100-disk-1@rep_test100_2015-06-12_11:03:01 | ssh firstname.lastname@example.org zfs receive vm/vm-200-disk-1 root@zfs2:~# ssh email@example.com root@zfs1:~# cp /var/lib/pve-zsync/100.conf.rep_test100_2015-06-11_14:11:01 /etc/pve/qemu-server/200.conf root@zfs1:~# nano /etc/pve/qemu-server/200.conf
Now you have to change the storage path:
bootdisk: virtio0 cores: 1 memory: 512 name: Debian8min net0: virtio=12:5E:F6:59:A9:BB,bridge=vmbr0 numa: 0 ostype: l26 smbios1: uuid=11fa2fba-5670-4610-aabb-534ad7edeffe sockets: 1 virtio0: zfs:vm-100-disk-1,size=10G virtio1: zfs:vm-100-disk-2,size=10G
bootdisk: virtio0 cores: 1 memory: 512 name: Debian8min net0: virtio=12:5E:F6:59:A9:BB,bridge=vmbr0 numa: 0 ostype: l26 smbios1: uuid=11fa2fba-5670-4610-aabb-534ad7edeffe sockets: 1 virtio0: vm:vm-200-disk-1,size=10G virtio1: vm:vm-200-disk-2,size=10G
Don't forget the command only works with IP Addresses and not hostnames.
Job status is on error and data-set can't be erased on destination system
If you have problems with a sync-job and when you try to erase the destination zvol, you get the error "zfs data-set is busy", then lvm could be the problem.
This can be occur if you sync zvols with a lvm on it.
In this case please insert the following line in /etc/lvm/lvm.conf
filter = [ "r|/dev/zd*|" ]
and reboot the system.
- as of 2016-11 If you migrate a vm that is used by pve-zsync
- edit /etc/cron,d/pve-zsync and change the IP address for vm .
- run 'pve-zsync enable --source ____ --name ____ ' or else there will be a warning when the job runs.