Brioche Backup
Table of contents
1 Abstract
Brioche is yet another backup shell script. Its main features are
- Full and differential backups
- LVM snapshots
- Xen oriented (somehow)
- Upload to a distant FTP server.
2 Rationale
Large numbers of backup solutions are freely available today, but when playing with incremental or differential backups, most of them rely on some filesystem capabilities, like hard-linking. If the only remote location available to store the backups is an FTP server, this is not a solution. Brioche relies on GNU tar's --listed-incremental option to create real differential [1] archives.
A second feature that makes Brioche interresting is the ability to use LVM snapshots. In the case of a Xen setup where the domUs use logical volumes as partitions, Brioche is able to backup everything from the dom0.
[1] | Each differential backup is based upon the last full backup, unlike incremental, which are based upon the last (full or incremental) backup. |
3 Getting Brioche
Download brioche-1.1.tar.gz
You can download the tarball from http://www.alrj.org/pages/brioche.html or get the latest development version with the following git command:
git clone http://git.alrj.org/git/brioche.git
A gitweb interface is also available at http://git.alrj.org/
4 Requirements
Brioche relies on a few easily available free software :
- In any case, Brioche will require GNU tar, which is able to deal with incremental backups. Don't even try it with any other tar implementation. Tested with version 1.16.
- For LVM snapshots, lvm2 will obviously be needed.
- FTP backups, if they're used, will require lftp.
See the References section for links to the aforementioned softwares.
5 Installation
Copy the three files brioche, brioche.conf and briochetab where you like, and set the CONFIG_FILE variable in the brioche script accordingly. If needed, run chmod +x /path/to/brioche. In a typical setup, the brioche script will be put in /usr/local/bin with the other two files under /etc.
6 Using Brioche
6.1 Configuration
Edit the file brioche.conf to suit your needs. Each option is commented inline and will be detailed here.
- BACKUPTAB
- Full absolute path to the briochetab file. This file describes which logical volumes and partitions must be backed up. Its format is explained in the Defining backups section.
- MAILTO
- When the job is done, Brioche will send a summary of the operations by email. You can set it to any value that your mail command understands.
- REPODIR
- Brioche will store all the generated archives in this directory.Note that before doing a full backup, Brioche will move all previous archives into an "undo" subdirectory, which will be removed only if the backup is successful. Make sure there's enough free space on the device where REPODIR is located.
- USAGE_WARN
- The report email will include a warning if the space used on REPODIR goes beyond the given threshold. The value must be an integer.
- COMPRESS
- This directive allows to specify the compression method to apply to the archives. Possible values are "none", "gz", "bz2" and "lzma". Warning : lzma may not be available with older versions of GNU tar.
- TAR_OPTS
- Additionnal options that you may want to pass to tar. A typical value could be "--one-file-system -S". The first option will skip all other mointpoints (very usefull if you have /dev, /proc, /sys or REPODIR mounted under a device that must be archived). The second one will try to deal with sparse files.
- SNAPSHOT_MOUNTPOINT
- Sets the directory where the temporary LVM snapshots must be mounted.
- SNAPSHOT_NAME
- The name to use for the snapshot volumes.
- SNAPSHOT_SIZE
- Set the size of the snapshot volume. The same suffix than for lvcreate(8) are available.
- USE_FTP
- If set to "yes", Brioche will upload the backups on an FTP server. See the Using FTP section for more information about this feature.
- FTP_HOST
- The address of the FTP server.
- FTP_DIR
- The base directory on the FTP server under which all the archives will be stored. Brioche will never touch anything that is not below this directory.
- FTP_KEEP
- Tells Brioche to keep a certain amount of older runs on the FTP. A run is a full backup plus all its subsequent differential backups. See the Using FTP section for a more detailed explanation.
6.2 Defining backups
The backups are defined in the file birochetab. Here is a typical example for a Xen config where cottman is the dom0 and syrtis, kadarin, valeron are domUs:
# Partition or LV Snapshot Host name Volume name # --------------------------------------------------------------- / no cottman root /usr no cottman usr /dev/vg00/valeron-root yes valeron root /dev/vg00/kadarin-root yes kadarin root /dev/vg00/kadarin-home yes kadarin home /dev/vg00/syrtis-root yes syrtis root /dev/vg00/syrtis-home yes syrtis home /dev/vg00/syrtis-usr yes syrtis usr /dev/vg00/syrtis-var yes syrtis var
Blank lines, or lines beginning with # are ignored.
The first column defines the directory or logical volume to backup. In this example,the first two lines are plain directories, while the other ones point to LVM devices.
The second column specify if the backup should be taken from an LVM snapshot or not. It must be set to "no" for the backup of a directory and to "yes" for the backup of a logical volume.
The last two columns are more or less cosmetic, and define where the archive files will be stored, and how they'll be named. The destination directory will be created under the REPODIR, and its name will be the value on the third column. Inside this directory, archive files will be named from the value given in the fourth column. Grouping by domUs' hostnames is only a suggestion, it can be completely different and adapted to suit your needs.
For instance, a full and a differential backup for the host valeron of the previous example would lead to the following structure:
user:/REPODIR$ ls -l valeron/ total 356544 -rw-r--r-- 1 root root 363545613 Jan 3 03:10 root.full.20090103.tar.bz2 -rw-r--r-- 1 root root 504722 Jan 3 03:10 root.full.snar -rw-r--r-- 1 root root 504725 Jan 4 04:06 root.incr.20090104.snar -rw-r--r-- 1 root root 160542 Jan 4 04:06 root.incr.20090104.tar.bz2
WARNING ! In case your REPODIR is not on a distinct device, it will be included in the backup if you've included the device in your briochetab. To avoid this issue, you can either exclude it explicitely by adding an "--exclude=..." option in TAR_OPTS, or simply specify in your briochetab file which directories need to be archived. |
6.3 Using FTP
With the help of lftp, Brioche is able to store an history of backups on an FTP server. This is mainly usefull when no other distant repository is available. If possible, consider using an CIFS, NFS, sshfs or any other kind of remote mountpoint for your REPODIR.
The archives present in the local REPODIR will be mirrored to the FTP server after each backup, be it a full or a differential one.
Since there is no way to hide the credentials if they are passed to lftp on the command line, the authentication relies on your .netrc file. See man netrc(5) for more information. In the home directory of the user that runs Brioche (typically root's), create the .netrc file with the following lines:
machine ftp.example.com login username password SikRet
Don't forget to secure it with chmod 600 .netrc or lftp will refuse to use it. The machine name must match the FTP_HOST configuration directive in brioche.conf.
On the FTP server, Brioche will keep a configurable amount of runs. Each run consists of a full backup and all the differential backups that are based on it. Before doing a full backup, Brioche will rotate the runs and keep only the configured number of older backups. The current backups can always be found under /FTP_DIR/hostname/latest/. Older ones will be under /FTP_DIR/hostname/run-X/ with X equal to 1 for the previous run, 2 for the one before and so forth, up to the value of FTP_KEEP.
Here's what happens during the rotation:
- the oldest run is removed
- all the run-X/ directories are shifted (run-3/ becomes run-4/, etc)
- the latest/ directory is renamed to run-1/
- a new, empty, latest/ directory is created, ready to accept the new files.
6.4 Running Brioche
Brioche understands the following arguments:
-f, --full | Do a full backup (by default, brioche will try to do a differential). |
-h, --help | Show a very limited help. |
When everything is ready, execute the brioche script as root. The script is pretty verbose, so don't panic if you see lots of lines scrolling in your terminal. By default, Brioche will try to make differential backups, but will gracefully fall back and do a full backup if none is available.
If all is fine, it can be added in the system crontab. Here's a suggestion for weekly full backup on Sunday, with differential during the weekdays:
# Daily incremental backup 30 3 * * 1-6 /usr/local/bin/brioche > /var/log/backup.`dow`.log 2>&1 # Weekly full backup on Sunday 30 3 * * 0 /usr/local/bin/brioche -f > /var/log/backup.`dow`.log 2>&1
The output of the script will be saved in /var/log/backup.DOW.log with DOW being the abbreviated day of the week (see man date(1) for the format). The dow bash script is provided alongside Brioche.
7 Bug reporting
There's no bugtracker for this project, your bug reports should be sent to the author : Amand Tihon <amand.tihon@alrj.org>. Please include as much information as possible in your report.
8 References
- GNU tar documentation : http://www.gnu.org/software/tar/manual/
- LVM documentation and links : http://sourceware.org/lvm2/
- lftp homepage : http://lftp.yar.ru/