Fischglas

You've loaded an old revision of the document! If you save it, you will create a new version with this data.

**CAVEAT : WORK IN PROGRESS **

====== rsnapshot zfs helper ======

This is a plugin for rsnapshot (and of course rsnapshot-backup) to be used with zfs as storage backend.

===== general idea =====

The main objective why you may consider using zfs might be:

rsnapshot begins its job with creating a directory filled with hard links as "copy" of yesterday's backup,
which subseqently is updated with rsync. This '''cp -al daily.1 daily.0''' takes a significant amount of time,
around 10-20% of the total time the backup takes. Worse, it creates a lot of disk activity, putting stress
on hard drives, blocking concurrent backups and creating high loads on the backup system. To some extend 
the deletion of expired backups will add to this overhead.

As zfs uses copy-on-write there is no need to somehow emulate this with '''cp -al ...'''.

===== example for rsnapshot ======

[[project:rsnapshot-zfs-helper:rsnapshot-example|Example for using rsnapshot-zfs-helper with rsnapshot]]

===== example for rsnapshot-backup ======

[[project:rsnapshot-zfs-helper:rsnapshot-backup-example|Example for using rsnapshot-zfs-helper with rsnapshot-backup]]

===== setup using rsnapshot-backup =====

==== preface ====

I'm using ''rsnapshot-backup'' which is basically a wrapper around rsnapshot.

It uses one setup per system to backup with one config, exclude and script file.
There are also individual backup and logging directories.
There is a scheduler which parallelizes backup runs and a configuration frontend
using template files to create the backup jobs.

There is also an enabling mechnism using symlinks.

A typical backup job configuration consists of:

<code>
/etc/rsnapshot-backup.d/conf.<BACKUP-CLIENT>
/etc/rsnapshot-backup.d/exclude.<BACKUP-CLIENT>
/etc/rsnapshot-backup.d/script.<BACKUP-CLIENT>    # optional
/etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> -> ../conf.<BACKUP-CLIENT>

/backup/hosts/<BACKUP-CLIENT>/daily.0
/backup/hosts/<BACKUP-CLIENT>/...
</code>

Here, this all runs on Debian GNU/Linux, so the examples refer to this environment.

==== details ====

rsnapshot-zfs-helper is to be configured inside rsnapshot's configuration files. i.e. '''/etc/rsnapshot-backup.d/conf.<BACKUP-CLIENT>'''

Additionally, '''cmd_cp''' has to be set to '''/bin/true''' - so the "cp -al ..." is rendered harmless.

<code>
cmd_cp        /bin/true
cmd_preexec   /usr/sbin/rsnapshot-zfs-helper /etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> preexec
cmd_postexec  /usr/sbin/rsnapshot-zfs-helper /etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> postexec
</code>

Example for creating a zpool:
<code>
##
## setup zfs (optional)
##
# 'bpo' kernel won't do the trick
apt-get install linux-image-amd64
apt-get clean
reboot

- add "contrib" to the apt-sources
apt-get update
apt-get install zfs-dkms zfsutils-linux
apt-get clean

/sbin/modprobe zfs

## create partition for zfs:
Device     Boot   Start       End   Sectors  Size Id Type
...
/dev/sda3       4194304 209715199 205520896   98G bf Solaris

## create zpool without mount point
zpool create zp-bak /dev/sda3 -m none

# create prereq. datasets
zfs create zp-bak/backup                        -o compression=lz4 -o snapdir=visible -o xattr=sa -o acltype=posixacl -o mountpoint=/backup
zfs create zp-bak/backup/hosts                  -o compression=lz4 -o snapdir=visible -o xattr=sa -o acltype=posixacl -o mountpoint=/backup/hosts
df  
->
   Filesystem     1K-blocks    Used Available Use% Mounted on
   ...
   zp-bak/backup        99041152     128  99041024   1% /backup
   zp-bak/backup/hosts  99041152     128  99041024   1% /backup/hosts
</code>

Create a zfs dataset for a single backup:
<code>
zfs create <ZPOOL>/backup/hosts/<BACKUP-CLIENT> -o compression=lz4 -o snapdir=visible -o xattr=sa -o acltype=posixacl -o mountpoint=/backup/hosts/<BACKUP-CLIENT>/daily.0
# e.g.
zfs create zp-bak/backup/hosts/srv42 -o compression=lz4 -o snapdir=visible -o xattr=sa -o acltype=posixacl -o mountpoint=/backup/hosts/srv42/daily.0
</code>

Create cron jobs for backup promotion to higher rotation levels
<code>
45 23 * * * root /usr/sbin/rsnapshot-zfs-helper /etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> daily
50 23 * * 5 root /usr/sbin/rsnapshot-zfs-helper /etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> weekly
55 23 1 * * root /usr/sbin/rsnapshot-zfs-helper /etc/rsnapshot-backup.d/enabled/conf.<BACKUP-CLIENT> monthly
</code>

===== how it works =====

* '''/usr/sbin/rsnapshot-zfs-helper ... preexec''' removes expired backups (= destroy zfs snapshots)
  * the removal of expired backups using ''rm'' will never occur because there is only a '''daily.0''' as far as '''rsnapshot''' is concerned
  * for the same reason, there will be no renaming or rotation of previous backups
  * '''rsync''' updates '''daily.0''' as usual
  * '''/usr/sbin/rsnapshot-zfs-helper ... postexec''' creates a zfs snapshot using the name extension '''@daily.<ISO8601-DATE-UTC>'''

===== FAQ =====

* Why does '''rsnapshot-zfs-helper''' use timestamps instead of enumeration when naming the snapshots? **A:** Because I like it better that way: you see what you get. No renaming required. No moving targets. 
  * Why the weird ISO date format? **A:** because it's an international standard. Lots of code is available to handle it. It's human readable. It scales well. No built in misinterpretation.
  * But why then use UTC? **A:** I would have preferred to use ISO-8601 with timezone notation. But zfs does not accept '+' as a character in snapshot names. So I used the format closest by - the UTC timestamp.
  * What's the deal with '''cmd_cp /bin/true'''? **A:** Think of it as NOOP.
  * Why is the .zfs subdir hidden ('''snapdir=hidden''')? **A** Because rsync would traverse it and all the subsequent snapshots resulting in really bad performance.

Edit summary

Note: By editing this page you agree to license your content under the following license: CC Attribution-Share Alike 4.0 International