Quick Start

This guide will show you how to do a backup - scrub - restore - cleanup cycle with sqlite as metadata backend and a file storage backup target (e.g. NFS).

What you need to know:

Backup source

A block device or image file (e.g. containing a VM) to be backed up. backy2 can not backup folders or multiple files. The source must not be modified during backup, so either stop all writers or create a snapshot.

Backup target

A storage (currently supported: filesystem or S3 compatible) to which the backed up data will be saved in 4MB blocks.

Backup Metadata

A Database containing information on how to reassemble the stored blocks to get the original data back. Currently an SQL database.

Version

A version of a backup. A version is a backup on a specific time for a specific backup source. A version is identified by its UUID.

Installation

Ubuntu 18.04

Installation:

Get the latest release from https://github.com/wamdam/backy2/releases
sudo dpkg -i backy2_*.deb          # Install the debian archive
sudo apt-get -f install            # Install the dependencies

# if you're using rbd/rados from ceph, we recommend the newer libraries
sudo apt install python3-rbd python3-rados

# if you're going to use postgresql
sudo apt install python3-psycopg2

Edit backy.cfg:

vim /etc/backy.cfg

Especially look if these paths are good.

1. Metadata storage path

   engine: sqlite:////var/lib/backy2/backy.sqlite
   

2. Data storage path

   path: /var/lib/backy2/data  # This should be the mountpoint of NFS
   

Other values of interest are simultaneous_writes and simultaneous_reads. Depending on your backup source and target you may want to go lower (i.e. disk with very slow seeks) or higher (raid source or target, S3 target, …).

Tip

For reference, we use about the half the number of disks as value for simultaneous access. So if you have 40 OSDs in ceph/rbd on the backup source and a 20 disk raid 10 backup target (which makes only 10 parallel disks on writes), you’d use

simultaneous_reads: 20
simultaneous_writes: 5

Of course your mileage may vary.

These settings have great impact on the backup and restore performance. Higher values need a bit more RAM.

Encryption key

Important: Set a unique encryption key. backy2 will not work without one. Also make sure it’s a good key. Generate it via:

$ openssl rand -hex 32

Set it to the config in the [DEFAULTS] Section:

encryption_key: decafbaddecafbaddecafbzddecafbaddecafbaddecafbaddecafbaddecafba

Note

The above key is intentionally invalid so that nobody copy&pastes this. The resulting key must be 64 hex characters long (which are effectively 32 bytes for the resulting AES key).

backup

1. Initialize the database:

   $ backy2 initdb
       INFO: $ /usr/bin/backy2 initdb
       INFO: Backy complete.
   

   Note

   Initializing a database multiple times does not destroy any data, instead will fail because it finds already-existing tables.

2. Create demo data:

   For demonstration purpose, create a 40MB test file:

   $ dd if=/dev/urandom of=TESTFILE bs=1M count=40
   40+0 records in
   40+0 records out
   41943040 bytes (42 MB, 40 MiB) copied, 0.175196 s, 239 MB/s
   

3. Backup the image (works similar with a device):

   $ backy2 backup file://TESTFILE myfirsttestbackup
       INFO: $ /usr/bin/backy2 backup file://TESTFILE myfirsttestbackup
       INFO: Backed up 1/10 blocks (10.0%)
       INFO: Backed up 2/10 blocks (20.0%)
       INFO: Backed up 3/10 blocks (30.0%)
       INFO: Backed up 4/10 blocks (40.0%)
       INFO: Backed up 5/10 blocks (50.0%)
       INFO: Backed up 6/10 blocks (60.0%)
       INFO: Backed up 7/10 blocks (70.0%)
       INFO: Backed up 8/10 blocks (80.0%)
       INFO: Backed up 9/10 blocks (90.0%)
       INFO: Backed up 10/10 blocks (100.0%)
       INFO: New version: 8fd42f1a-2364-11e7-8594-00163e8c0370 (Tags: [b_daily,b_weekly,b_monthly])
       INFO: Backy complete.
   

4. List backups:

   $ backy2 ls
       INFO: $ /usr/bin/backy2 ls
   +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+
   |         date        | name              | snapshot_name | size | size_bytes |                 uid                  | valid | protected | tags                       |   expire   |
   +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+
   | 2017-04-17 11:54:07 | myfirsttestbackup |               |   10 |   41943040 | 8fd42f1a-2364-11e7-8594-00163e8c0370 |   1   |     0     | b_daily,b_monthly,b_weekly | 2019-11-11 |
   +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+
       INFO: Backy complete.
   

scrub

Scrubbing reads all the blocks from the backup target (or some of them if you use the -p option) and compares them with the metadata or, if you pass a source option (-s), also with the original data.

$ backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370
    INFO: $ /usr/bin/backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370
    INFO: Backy complete.

If an error occurs (for example, if backup target blocks couldn’t be read or data has changed), this looks like that:

$ backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370
     INFO: $ /usr/bin/backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370
    ERROR: Blob not found: Block(uid='2c0910bef8qnAm54mnyBsonRsPBfTzP', version_uid='8fd42f1a-2364-11e7-8594-00163e8c0370', id=8, date=datetime.datetime(2017, 4, 17, 11, 54, 7, 639022), checksum='41c9aa8df42913b3544270a10f1b219cd1b5e1ad9d51700e97acdaeaed3cea843ffaad99590e07de260918ce3847a8b612c9f51f2c945a2d14238956a49ca572', size=4194304, valid=1)
     INFO: Marked block invalid (UID 2c0910bef8qnAm54mnyBsonRsPBfTzP, Checksum 41c9aa8df42913b3544270a10f1b219cd1b5e1ad9d51700e97acdaeaed3cea843ffaad99590e07de260918ce3847a8b612c9f51f2c945a2d14238956a49ca572. Affected versions: 8fd42f1a-2364-11e7-8594-00163e8c0370
     INFO: Marked version invalid (UID 8fd42f1a-2364-11e7-8594-00163e8c0370)
    ERROR: Marked version invalid because it has errors: 8fd42f1a-2364-11e7-8594-00163e8c0370

The exit code is then != 0. Which one exactly depends on the kind of error:

$ echo $?
20

Also, the version is marked invalid as you can see in:

$ backy2 ls
    INFO: $ /usr/bin/backy2 ls
+---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+
|         date        | name              | snapshot_name | size | size_bytes |                 uid                  | valid | protected | tags                       |
+---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+
| 2017-04-17 11:54:07 | myfirsttestbackup |               |   10 |   41943040 | 8fd42f1a-2364-11e7-8594-00163e8c0370 |   0   |     0     | b_daily,b_monthly,b_weekly |
+---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+
    INFO: Backy complete

Just in case you are able to fix the error, just scrub again and backy2 will mark the version valid again.

restore

Restore into a file or device:

$ backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE
    INFO: $ /usr/bin/backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE
    INFO: Restored 1/10 blocks (10.0%)
    INFO: Restored 2/10 blocks (20.0%)
    INFO: Restored 3/10 blocks (30.0%)
    INFO: Restored 4/10 blocks (40.0%)
    INFO: Restored 5/10 blocks (50.0%)
    INFO: Restored 6/10 blocks (60.0%)
    INFO: Restored 7/10 blocks (70.0%)
    INFO: Restored 8/10 blocks (80.0%)
    INFO: Restored 9/10 blocks (90.0%)
    INFO: Restored 10/10 blocks (100.0%)
    INFO: Backy complete.

backy2 prevents you from restoring into an existing file (or device). So if you try again, it will fail:

$ backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE
    INFO: $ /usr/bin/backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE
   ERROR: Target already exists: file://RESTOREFILE
Error opening restore target. You must force the restore.

If you want to overwrite data, you must --force.

Note

For more (and possibly faster) restore methods, please refer to the section Restore.

expire

Usually you know when to expire a backup at backup time. So when creating a backup you may also add the expiration date:

$ backy2 backup file://tmp/test test -e 2020-01-24

The expiration date can be shown in backy2 ls and you may also show expired versions only via backy2 ls -e.

version rm and cleanup

You can remove any given backup version by:

$ backy2 rm 8fd42f1a-2364-11e7-8594-00163e8c0370
    INFO: $ /usr/bin/backy2 rm 8fd42f1a-2364-11e7-8594-00163e8c0370
   ERROR: Unexpected exception
   ERROR: 'Version 8fd42f1a-2364-11e7-8594-00163e8c0370 is too young. Will not delete.'
   […]
    INFO: Backy failed.

What prevents this version to be deleted is the backy.cfg option

disallow_rm_when_younger_than_days: 6

However, instead of changing this option, you can simply use the --force:

$ backy2 rm 8fd42f1a-2364-11e7-8594-00163e8c0370 --force
    INFO: $ /usr/bin/backy2 rm 8fd42f1a-2364-11e7-8594-00163e8c0370 --force
    INFO: Removed backup version 8fd42f1a-2364-11e7-8594-00163e8c0370 with 10 blocks.
    INFO: Backy complete.

backy2 stores each block in the backup target (i.e. filesystem, s3, …) once. If it encounters another block on the backup source with the same checksum 1, it will only write metadata which refers to the same backup target block.

So if a backup is deleted, backy2 needs to check if all references to each block are gone. So backy2 can’t just simply wipe all blocks from a removed backup version.

As this is a resource-intensive task, it’s separated into a special command:

$ backy2 cleanup
    INFO: $ /usr/bin/backy2 cleanup
    INFO: Cleanup-fast: Cleanup finished. 0 false positives, 0 data deletions.
    INFO: Backy complete.

As you can see, nothing has been deleted. The reason for this is that backy2 is prepared to be used in parallel (backups during restores during scrubs). On some edges (and this is one), it has timeouts to let data which might be in the flow or in caches settle.

For cleanup this timeout is 1 hour. So you can now wait for 1 hour and repeat, or you can use the alternative cleanup option (-f) which will ignore this timeout. However be warned (also shown when doing a backy2 cleanup --help):

Note

A full cleanup must not be run parallel to ANY other backy jobs. backy2 will prevent you from doing this by creating a global lock on the backup server.

Caution

Parallelism has been tested successfully with PostgreSQL. It might not work reliably with other DBMS.

1

backy2 uses sha512 which can be configured in backy.cfg. sha512 is the only recommended checksum, however it’s possible to use any other algorithm from python3’s hashlib (i.e. md5, sha1, sha224, sha256, sha384).