💾 Archived View for perso.pw › blog › articles › qubes-os-efficient-backup.gmi captured on 2023-06-14 at 14:00:01. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-05-24)
-=-=-=-=-=-=-
These days, I've been playing a lot with Qubes OS, it has an interesting concept of deploying VMs (using Xen) in a well integrated and transparent manner in order to hardly separate every tasks you need.
By default, you get default environments such as Personal, Work and an offline Vault, plus specials VMs to handle USB proxy, network and firewall. What is cool here is that when you run a program from a VM, only the window is displayed in your window manager (xfce), and not the whole VM desktop.
The cool factor with this project is their take on the real world privacy and security need, allowing users to run what they need to run (proprietary software, random binaries), but still protect them. Its goal is totally different from OpenBSD and Tails. Did I say you can also route a VM network through Tor out of the box? =D
If you want to learn more, you can visit Qubes OS website (or ask if you want me to write about it):
New user guide: How to organize your cubes (nice reading to understand Qubes OS)
If you know me, you should know I'm really serious about backups. This is incredibly important to have backups.
Qubes OS has a backup tool that can be used out of the box, it just dump the VMs storage into an encrypted file, it's easy but not efficient or practical enough for me.
If you want to learn more about the format used by Qubes OS (and how to open them outside of Qubes OS), they wrote some documentation:
Qubes OS: backup emergency restore
Now, let's see how to store the backups in Restic or Borg in order to have proper backups.
/!\ While both software support deduplication, this doesn't work well in this case because the stored data are compressed + encrypted already, which has a very high entropy (it's hard to find duplicated patterns).
Qubes OS backup tool offers compression and encryption out of the box, but when it comes to the storage location, we can actually use a command to send the backups to the command's stdin, and guess what, both restic and borg support receiving data on their standard input!
I'll demonstrate how to proceed both with restic and borg with a simple example, I recommend to build your own solution on top of it the way you need.
Screenshot of Qubes backup tool
As we are running Qubes OS, I prefer to create a dedicated backup VM using the Fedora template, it will contain the passphrase to the repository and an SSH key for remote backup.
You need to install restic/borg in the template to make it available in that VM.
If you don't know how to install software in a template, it's well documented:
Qubes OS: how to install software
Generate an SSH key if you want to store your data on a remote server using SSH, and deploy it on the remote server.
In order to simplify the backup command configuration in the backup tool (it's a single input line), but don't sacrifice on features like pruning, we will write a script on the backup VM doing everything we need.
While I'm using a remote repository in the example, nothing prevents you from using a local/external drive for your backups!
The script usage will be simple enough for most tasks:
Write a script in `/home/user/restic.sh` in the backup VM, it will allow simple customization of the backup process.
#!/bin/sh
export RESTIC_PASSWORD=mysecretpass
# double // is important to make the path absolute
export RESTIC_REPOSITORY=sftp://solene@10.42.42.150://var/backups/restic_qubes
KEEP_HOURLY=1
KEEP_DAYS=5
KEEP_WEEKS=1
KEEP_MONTHS=1
KEEP_YEARS=0
case "$1" in
init)
restic init
;;
list)
restic snapshots
;;
restore)
restic restore --target . $2
;;
backup)
cat | restic backup --stdin
restic forget \
--keep-hourly $KEEP_HOURLY \
--keep-daily $KEEP_DAYS \
--keep-weekly $KEEP_WEEKS \
--keep-monthly $KEEP_MONTHS \
--keep-yearly $KEEP_YEARS \
--prune
;;
esac
Obviously, you have to change the password, you can even store it in another file and use the according restic option to load the passphrase from a file (or from a command). Although, Qubes OS backup tool enforces you to encrypt the backup (which will be store in restic), so encrypting the restic repository won't add any more security, but it can add privacy by hiding what's in the repo.
/!\ You need to run the script with the parameter "init" the first time, in order to create the repository:
$ chmod +x restic.sh $ ./restic.sh init
Write a script in `/home/user/borg.sh` in the backup VM, it will allow simple customisation of the backup process.
#!/bin/sh
export BORG_PASSPHRASE=mysecretpass
export BORG_REPO=ssh://solene@10.42.42.150/var/solene/borg_qubes
KEEP_HOURLY=1
KEEP_DAYS=5
KEEP_WEEKS=1
KEEP_MONTHS=1
KEEP_YEARS=0
case "$1" in
init)
borg init --encryption=repokey
;;
list)
borg list
;;
restore)
borg extract ::$2
;;
backup)
cat | borg create ::{now} -
borg prune \
--keep-hourly $KEEP_HOURLY \
--keep-daily $KEEP_DAYS \
--keep-weekly $KEEP_WEEKS \
--keep-monthly $KEEP_MONTHS \
--keep-yearly $KEEP_YEARS
;;
esac
Same explanation as with restic, you can save the password elsewhere or get it from a command, but Qubes backup already encrypt the data, so the repo encryption will mostly only add privacy.
/!\ You need to run the script with the parameter "init" the first time, in order to create the repository:
$ chmod +x borg.sh $ ./borg.sh init
Now, configure the Qubes backup tool:
While it's nice to have backups, it's important to know how to use them. The setup doesn't add much complexity, and the helper script will ease your life.
On the backup VM, run `./borg.sh list` (or the restic version) to display available snapshots in the repository, then use `./borg.sh restore $snap` with the second parameter being a snapshot identifier listed in the earlier command.
You will obtain a file named `stdin`, this is the file to use in Qubes OS restore tool.
If you don't always backup all the VMs, if you keep the retention policy like in the example above, you may lose data.
For example, if you have a KEEP_HOURLY=1, create a backup of all your VMs, and just after, you specifically want to backup a single VM, you will lose the previous full backup due to the retention policy.
In some cases, it may be better to not have any retention policy, or simply time based (keep snapshots which date < n days).
Using this configuration, you get all the features of a industry standard backup solution such as integrity check, retention policy or remote encrypted storage.
In case of an issue with the backup command, Qubes backup will display a popup message with the command output, this helps a lot debugging problems.
An easy way to check if the script works by hand is to run it from the backup VM:
echo test | ./restic.sh backup
This will create a new backup with the data "test" (and prune older backups, so take care!), if it doesn't work this is a simple way to trigger a new backup to solve your issue.