PVE-zsync
Introduction
With the Proxmox VE ZFS replication manager (pve-zsync) you can synchronize your virtual machine (virtual disks and VM configuration) or directory stored on ZFS between two servers. By synchronizing, you have a full copy of your virtual machine on the second host and you can start your virtual machines on the second server (in case of data loss on the first server).
By default, the tool is syncing every 15 minutes, but the synchronization interval is fully configurable via the integrated cron job setup. The configuration of pve-zsync can be done either on the source server or on the target server.
This is useful for advanced backup strategies.
Note: pve-zsync is introduced for Proxmox VE 3.4 as technology preview. The package can be installed on plain Debian Wheezy or Jessie servers too, as long as ZFS is configured.
Note: Our ZFS is configured to automount all subvols so keep in mind if you use pve-szync and also zvol will be scanned by lvm.
Main features
- Speed limiter
- Syncing interval can be set by cron
- Syncing VM (disks and config) but also ZFS Datasets
- Can keep multiple backups
- Can be used in both directions
- Can send on local host
- Traffic is encrypted
Limitations
- not possible to sync recursive
- only ssh for transfer
- email notification is done by cron
- Name resolution not taken into account, you have to use IP Addresses, even using hosts file hostnames will not work
System requirements
- Both, target and source server must support ZFS (best practice: use Proxmox VE hosts).
- SSH must be installed and configured
- to receive email-notifications, a working mail server is required (e.g. postfix).
- cstream
- perl
- scp
- JASON.pm (libjson-perl this is the Debian package name, e.g. needed on Wheezy)
Configuration and use
Install the package with apt, on your Proxmox VE host:
apt-get install pve-zsync
This tool need basically no configuration. On the the first usage, when you create a job with a unknown host, the tool will ask you about the password for the remote server.
Sync a VM or ZFS dataset one time
(N.B. this is also possible if a recurring job for that VM already exists, here you must have in mind that the naming in --name must be the same).
root@zfs1:~# pve-zsync sync --source 100 --dest 192.168.1.2:tank/backup --verbose --maxsnap 2 --name test1 --limit 512
This command sync VM 100 witch is located on the server where the tool is called and sent it to the server 192.168.1.2 on the zpool tank which has an subset backup. The --maxsnap tells that it should be kept 2 backups, if there are more then 2 backups, the 3 one will be erased (sorted by creation time). Name is only needed if there is already a sync job. The --limit param sets the speed limit what is used for syncing, here it would be 512 KBytes/s.
Create a recurring sync job
root@zfs2:~# pve-zsync create --source 192.168.1.1:100 --dest tank/backup --verbose --maxsnap 2 --name test1 --limit 512 --skip
The --skip parameter disables the initial sync, that normally would be done immediately but can take a while, depending on the size of the backup. The initial sync will be done at the first sync time.
Delete a recurring sync job
if you delete a job, the former backup data will not be erased only the config will be erased.
root@zfs2:~# pve-zsync destroy --source 192.168.1.1:100 --name test1
Name is not necessary if it is default.
Pause a sync job
If you want to pause a job, say maintenance the source server!
root@zfs2:~# pve-zsync disable --source 192.168.1.1:100 --name test1
Reactivate a sync job
to reactivate a job, because the job was paused or the job failed use.
root@zfs2:~# pve-zsync enable --source 192.168.1.1:100 --name test1
This will reset the error flag in case of failure.
Changing parameters
You can edit the configuration in /etc/cron.d/pve-zsync or destroy the job and create it new.
Information about the jobs
To get a overview about all jobs use:
root@zfs94:~# pve-zsync list SOURCE NAME STATE LAST SYNC TYPE 192.168.15.95:100 testing1 ok 2015-05-13_14:44:00 ssh 192.168.15.95:data/test1 testing1 syncing2015-05-13_14:44:11 ssh root@zfs94:~# pve-zsync status SOURCE NAME STATUS 192.168.15.95:100 testing1 ok 192.168.15.95:data/test1 testing1 syncing
Recovering an VM
You must recover the VM or dataset manually. (In one of the upcoming releases, the restore for Proxmox VE VM will be integrated).
Fist, stop the sync job for the VM or dataset in question.
NOTE: if not you can be interfere the sync job or your snapshot will removed before you are able to send it.
root@zfs2:~# pve-zsync disable --source 192.168.15.1:100 --name test root@zfs2:~# pve-zsync list SOURCE NAME STATE LAST SYNC TYPE 192.168.15.1:100 test stopped2015-06-12_11:03:01 ssh
Then you can send the VM or Dataset to the selected target. SSH is only needed if you send to a remote sever.
zfs send <pool>/[<path>/]vm-<VMID>-disk-<number>@<last_snapshot> | [ssh root@<destination>] zfs receive <pool>/<path>/vm-<VMID>-disk-<number>
If you have a VM you must also copy the config and you need to correct the virtual disk storage configuration accordingly.
cp /var/lib/pve-zsync/<VMID>.conf.rep_<JOB_NAME><VMID>_<TIMESTAMP> /etc/pve/qemu-server/<VMID>.conf
Note: On PVE 4.2+ the path is: /var/lib/pve-zsync/data/<VMID>.conf.rep_<JOB_NAME><VMID>_<TIMESTAMP>
Example restore VM 100 with 2 disk from 192.168.15.2 (pve2) to 192.168.15.1 (pve1) and change VMID from 100 to 200:
root@zfs2:~# zfs send rpool/backup/vm-100-disk-1@rep_test100_2015-06-12_11:03:01 | ssh root@192.168.15.1 zfs receive vm/vm-200-disk-1 root@zfs2:~# ssh root@192.168.15.1 root@zfs1:~# cp /var/lib/pve-zsync/100.conf.rep_test100_2015-06-11_14:11:01 /etc/pve/qemu-server/200.conf root@zfs1:~# nano /etc/pve/qemu-server/200.conf
Now you have to change the storage path:
bootdisk: virtio0 cores: 1 memory: 512 name: Debian8min net0: virtio=12:5E:F6:59:A9:BB,bridge=vmbr0 numa: 0 ostype: l26 smbios1: uuid=11fa2fba-5670-4610-aabb-534ad7edeffe sockets: 1 virtio0: zfs:vm-100-disk-1,size=10G virtio1: zfs:vm-100-disk-2,size=10G
to
bootdisk: virtio0 cores: 1 memory: 512 name: Debian8min net0: virtio=12:5E:F6:59:A9:BB,bridge=vmbr0 numa: 0 ostype: l26 smbios1: uuid=11fa2fba-5670-4610-aabb-534ad7edeffe sockets: 1 virtio0: vm:vm-200-disk-1,size=10G virtio1: vm:vm-200-disk-2,size=10G
Troubelshooting
Don't forget the command only works with IP Addresses and not hostnames.
Job status is on error and data-set can't be erased on destination system
If you have problems with a sync-job and when you try to erase the destination zvol, you get the error "zfs data-set is busy", then lvm could be the problem.
This can be occur if you sync zvols with a lvm on it.
In this case please insert the following line in /etc/lvm/lvm.conf
filter = [ "r|/dev/zd*|" ]
and reboot the system.
tbd.