Our server pr0vider carried out maintenance on 15 January 2020 but it corrupted some files in the process. If you notice anything out of the ordinary (partial files, pages not loading, that kind of shit) then let us know at:

Bigheck backup scripts for QEMU/KVM-based VMs (as well as the host lol)

Wazakindjes 532a093690 Added n0te how about after rest0ring, .qcow2 files (pr0lly sparse files in general doe) physically still only occupy what they had alloc8ed before -- e.g. 10 GB thin disk w/ 5 GB actually in use/allocated will only 0ccupy 5 GB on-disk after rest0rin [=[=[[=[=[=[==[=[=[== 2 months ago
conf 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
exclusions 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
.gitignore 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
LICENSE ad13eccf0c Ayy lmao first versi0ne [[==[==[=[[=[==[[==[ 2 months ago
README.md 532a093690 Added n0te how about after rest0ring, .qcow2 files (pr0lly sparse files in general doe) physically still only occupy what they had alloc8ed before -- e.g. 10 GB thin disk w/ 5 GB actually in use/allocated will only 0ccupy 5 GB on-disk after rest0rin [=[=[[=[=[=[==[=[=[== 2 months ago
backuppem.sh 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
list_restic_snapsh0ts.sh 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
list_restic_usage.sh ad13eccf0c Ayy lmao first versi0ne [[==[==[=[[=[==[[==[ 2 months ago
list_virsh_snapsh0ts.sh 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
run_backup_all.sh 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago
snapsh0ttem.sh 4a80a985d4 Added: mention of qemu-guest-agent for quiescing em guest file system prior to snapsh0ttin, readme section about helper scripts, mention not checking root privs, mention sparse/thin disk files still taking up (almost) all of the max size, some n0tes for viewing/working with backup data outside of the scripts, support for exclusion lists (both global and per path specification, see examples ;]), more flags for restic (e.g. tagging our backups and only forgetting backups matching that tag) || Changed: clarified B2 requirements, moved some ret=$? assignments below the actual command we wanted to check and not the following echo (fuckin kek), clarified disk specifier list in conf/backup.sh, unset all environment values every time the config is l0aded (before setting em 0fc), running of restic forget/prune/check post-backup is now configurable, do an IFS-based split for certain arrays due to otherwise splitting on spaces too, allow VM disk list to be empty (will just back up XML), fix support for multiple paths within the same array entry (using ("/home root") instead of ("/home" "/root") -- will share exclusions), removed --live flag from virsh snapshot-create-as because using it will only cause it to back up the memory state too (which is undesired) 2 months ago

README.md

The fuck is this

This is just a qt lil collection of scripts I made to back up mein QEMU/KVM-based VMs to Backblaze B2, encrypted and all.

Requirements

  • Obviously, a Backblaze B2 account and a bucket to store your shit in (make sure to set the lifecycle to Keep only the last version of the file, as we'll handle retention ourselves)
  • I'm using libvirt so if you're using plain QEMU or a different management layer, you prolly can't use this ;;;];];];]
  • The restic tool for actually writing that shit (does deduplication and encryption and shit =]]])
  • For VM snapsh0ts we'll always try to quiesce the file system (bringing it to a state suitable for backups, e.g. flushing memory buffers to disk), for this to work the guest must have the guest tools installed and configured (qemu-guest-agent package on l00nix)

Also, one of the components involved in creating snapsh0ts insists on creating one for every disk even if you only give it 1 disk specifier. To work around this, make sure that your VM's XML disks contain the snapshot='no' attribute: <disk type='file' device='disk' snapshot='no'>
This doesn't mean it won't ever create snapsh0ts, this is just the default action for any disk not explicitly specified in the snapsh0t command.

Usage

Go to the conf dir and copy the 2 *.sh.example files (remove .example 0bv m9). Everything is explained in thur.

There are 3 backup scripts in dis dere rep0:

  • snapsh0ttem.sh: a short wrapper script just for creating a snapsh0t for a VM
  • backuppem.sh: builds on snapsh0ttem.sh and backs up 1 disk for any given VM
  • run_backup_all.sh: a looper script incorporating both of these, it also backs up the host's files

I separated that shit because this way you can easily create individual snapsh0ts, or even back up an individual VM. [[==[=[[=[=

I also have a couple utility/helper scripts:

  • list_restic_snapsh0ts.sh: every backup target is called a snapsh0t in restic, this will simply list all of them (if you want to see all files in a snapsh0t: restic ls <snapsh0t ID>)
  • list_restic_usage.sh: lists the raw size of the backup repository, i.e. this is (close to) what Backblaze will charge you for ;]
  • list_virsh_snapsh0ts.sh: even though we use snapsh0ts without metadata, this script will list all snapsh0ts (of all VMs) currently known to libvirt

Muy importante: this shit doesn't check if you're running it as root nor does it elevate to it, it simply runs under the current user. Some people may have set up a separate management user that has permissions on everything they want/need backed up, so checking for rewt is pretty pointless imo tbh famalmalmala. ;]

Backup strategy

How VM backups are done depends on if they're currently running or not, but in all cases we always start with creating a snapsh0t. The reason for that is simple: even if a VM is currently not running, if something turns it on halfway through, your backup is most likely fucked if you simply copied the base disk file.

Muy importante: every disk has to be snapsh0tted separately, because I like to have that ability. ;] Also, if you're using thin-provisioned/sparse disks (only allocate bytes when needed) then restic does not (yet) skip the empty portion. I tried backing up a sparse .qcow2 file with a max size of 10 GB and only half actually allocated, when it was done Backblaze listed 9.1 GB and list_restic_usage.sh returned 8.47 GB. On the other hand, after restoring this disk du -h disk.qcow2 still reported about 5 GB in use. :DDDDDDDD

After the snapsh0t has been created, we simply use restic to copy the base disk file over to Backblaze. Once that's successful we can commit the snapsh0t back into the base disk and remove the snapsh0t, this is the part that's different based on VM state:

  • Running: using virsh blockcommit we can commit any changes while live, also the VM's XML config will be rewritten to point back to the original disk
  • Shut down: first we'll qemu-img commit to merge the changes, then with virt-xml --update we can "manually" rewrite the XML to point back to the original disk (there doesn't seem to be a feature within libvirt yet to easily commit offline snapsh0ts)

Then finally, the snapsh0t file is removed and it proceeds with backing up the host's files.

Some n0tes

  • If you want to delete a backup from restic, simply do restic forget <snapsh0t ID>. Keep in mind the data is only actually removed once you run restic prune. ;]
  • To view the full contents of a backup: restic ls <snapsh0t ID>