File System level backups with LVM snapshots

Introduction

The general idea consists in combining an external tool wich is able to do filesystem level incremental backups ( rsync by means of BackupPC in this document) with the possibility to take snapshots of LVM based storage of virtual machines.

Fundamental constraints in this solution are:

• Do not change fundamentally the configuration of an host under BackuPC
• Preserve easy interactive restore directly on the host.

Basically the target host, when a backup is required via ssh connection, instead of directly executing the rsync command, intercepts it and runs a script ("forced command") which:

1. Prepares backup operations (for instance, saving ACL in case of Windows host)
2. Stops or Suspends services which can do important changes on filesystem.
3. Triggers a snapshot of his own storage on PVE host it is runnng on.
4. Revert machine to normal operating state.
5. Redirects original rsync command towards PVE hosts and the snapshot.
   • Redirected rsync runs on PVE: mount fs, optionally save MBR and PBS, save ntfs metadata for Windows hosts, run rsync.
6. Triggers snapshot snapshot removal on PVE.

During interactive restore, instead, rsync process runs directly on the host.

In the following pargraphs detailed configuration steps for a Windows host are shown.

Requirements

• Local user "backup", member of Administrators and Backup Operators

Tools needed on target Windows host

• CYGWIN
  • sshd
  • rsync
  • iconv
  • subversion (used for script update)
  • vim
• SetACL
• Sync

Procedure

Create "backup" user

• Add backup user to "Administrators" and "Backup Operators groups."
• Connect to the host as "backup" user.
• If you have quotes activated for some disk, check that "backup" entry is "no limit" (interactively restored files are initially owned by this user.).

Install Cygwin as backup user

• Create C:\cygwin folder
  • Copy from another server c:\cygwin\cygwin-data folder (or install from the net if this is the first host configured).
  • Copy locally and run as backup the Cygwin install file "Setup.exe"
    • Install for all users
    • Let default setup for root cygwin folder.
    • Set local folder as repository and use c:\cygwin\cygwin-data as source.
  • Add following packages:
    • openssh
    • rsync (NOTE: install 3.0.7; 3.0.8 is problematic.)
    • libiconv
    • libiconv2
    • subversion
    • vim
  • Proceed, accepting Desktop and Start menu shortcuts.
  • Enter bash shell using Desktop icon; wait default settings creation for "backup" user; exit bash shell.

cyg_server user setup

NOTE: Skip this step for Windows XP hosts; in that case sshd will run with system account privileges.

• Reconnect to the host with a Domain Administrator account; enter bash, and run:

mkpasswd -l -d intra | grep cyg_server >> /etc/passwd

which adds in /etc/passwd cygwin file an entry for domain user cyg_server, ssh daemon will run with this user account.

• Add cyg_server to local Administrators.
• NOTE: It's important to check that cyg_server is listed as Domain Administrator in /etc/passwd, and that the same user is a local Administrator, before proceeding with following steps.

ssh service setup.

• Reconnect as local "backup" user.
• Run from bash "ssh-host-config" script; see in the following section the responses to various requests ("*** Query:" sections).

$ ssh-host-config

*** Query: Overwrite existing /etc/ssh_config file? (yes/no)  yes

*** Info: Creating default /etc/ssh_config file

*** Query: Overwrite existing /etc/sshd_config file? (yes/no) yes

*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.


*** Query: Should privilege separation be used? (yes/no) yes

*** Info: Note that creating a new user requires that the current account have
*** Info: Administrator privileges.  Should this script attempt to create a

*** Query:new local account 'sshd'? yes

*** Info: Updating /etc/sshd_config file


*** Warning: The following functions require administrator privileges!

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes 

*** Query: Enter the value of CYGWIN for the daemon: [] ntsec

*** Info: On Windows Server 2003, Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: You appear to be running Windows XP 64bit, Windows 2003 Server,
*** Info: or later.  On these systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd).

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.


*** Query: Do you want to use a different name? (yes/no) no

*** Info: Please enter a password for new user cyg_server.  Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.

*** Query: Please enter the password:
*** Query: Reenter:

*** Info: Also keep in mind that the user 'cyg_server' needs read permissions
*** Info: on all users' relevant files for the services running as 'cyg_server'.

*** Info: In particular, for the sshd server all users' .ssh/authorized_keys
*** Info: files must have appropriate permissions to allow public key
*** Info: authentication. (Re-)running ssh-user-config for each user will set
*** Info: these permissions correctly. [Similar restrictions apply, for
*** Info: instance, for .rhosts files if the rshd server is running, etc].

NOTE: In some cases (probably if you forgot to add cyg_server to local Administrators), errors like following could happen:

*** Warning: cyg_server is in /etc/passwd, but the local
*** Warning: machine's SAM does not know about cyg_server.
*** Warning: Perhaps cyg_server is a pre-existing domain account.
*** Warning: Continuing, but check if this is ok.

In that case, [wiki:howtoPreparareVmWinBackupPC#Verificapermessiutentecyg_server verify cyg_server permissions] as shown at the end of this document.

• Start ssh service

net start sshd

ssh client setup

• Copy from another backup host the ssh backup key

scp <hostname>:/home/backup/id_rsa_backup /home/backup/

All virtual machines connect to PVE hosts with the same key (generic access with this key is filtered with a forced command on PVE).

"Forced command" setup

Edit ~/.ssh/authorized keys, adding forced command for backup connections:

command="/home/backup/backup-restore" ssh-rsa AAAAB3Nza...

where AAAAB3Nza... is the remote backuppc public key.

host scripts

Install vm side scripts in /home/backup.

PVE scripts

• Install PVE side scripts in a directory of your choice (e.g.: /opt/snap)
• Configure forced command inside PVE authorized_keys" file:

command="/opt/snap/snap-backup-restore" ssh-rsa AAAAB3NzaC1...

where AAAAB3Nza... is the public counterpart of the id_rsa_backup configured on virtual machines.

NOTE: Take care that /opt/snap/snap-backup-restore is the correct location of the snapshot master script.

• You will need on PVE also a recent version of ntfs-3g (Debian Squeeze package is currently too old), and the attrib package (Squeeze verion is ok).