💾 Archived View for thrig.me › tech › openbsd › restic.gmi captured on 2023-09-28 at 17:28:14. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-09-08)
-=-=-=-=-=-=-
restic is backup software. It's not too terrible, but I haven't tried much else on OpenBSD, besides rsnapshot, which is older and slower and probably not recommended these days. This then is a basic restic usage guide (mostly for my own use) and probably too many random thoughts on backups (not for my own use).
restic can use all sorts of fancy things, though I only backup to an external USB device at the moment.
$ doas mkdir /backup
$ doas mount /backup
$ doas chmod 700 /backup
$ doas restic init -r /backup/restic
Probably you should save the password somewhere?
Also, it is easy to create a test repositories and practice with those, especially if you are new to the software and do not want to risk accidents with the real backups, like deleting all the backups or getting rid of all the passwords or some other horror story that would have been posted to alt.sysadmin.recovery back in the day.
$ doas restic -r /backup/restic key list
...
$ doas restic -r /backup/restic key add
...
$ doas restic -r /backup/restic key remove 810d210a
...
Not very fancy. Probably needs improvements, such as excludes.
#!/bin/sh
pushall
is-mounted /backup || doas mount /backup
doas restic -q -r /backup/restic backup /home /etc /var/backups
doas umount /backup
is-mounted /backup && echo 2>&1 "/backup still mounted??"
This is for an external USB device, which is kept unmounted by default. One reason the backup device is unmounted is to avoid a costly fsck if the filesystem goes down uncleanly. More importantly, stray commands are less likely to damage the contents of a filesystem that is not mounted except during backups and rare file recoveries. I don't often randomly type rm -rf as root, nor have the OpenBSD folks shipped scripts that do that (unlike, say, Apple, who also had a tradition of putting spaces in filenames) but defense in depth is a good thing, and nobody is perfect.
is-mounted and other utilities
If the system is maintained under configuration management then the /etc backup would likely not be necessary, on the presumption that configuration management would put the correct configuration in place on a newly built system. More of /var may need to be backed up, depending on what your system does.
I do not run any databases that would need special handling. Writes to a few sqlite databases for various things (gemini feed aggregation, a lojban dictionary, and the gemini blog) are rare and unlikely for me to run the same time as a backup. If you use different database software or have different write patterns then there may be a larger risk that backups cannot be done on a live database file.
If you need backup automation then you'll need to wrangle the password somehow. I'm pretty good at remembering to run the backups, usually near the end of the day, but they could also be run after doing something important. The backup script also runs the "pushall" script that finds git repositories that need pushing and pushes them: git can be another form of backup, especially if the bits get off to some other system. git is not so good on the encryption front by default, though this may not matter for repositories exposed to the public.
There are several other things you may want to backup: the current working kernel, especially if you customize it, the disklabel for any and all disks you have (these may be available as /var/backups/disklabel*), and maybe the bootloader section of the disk if you are doing anything fancy there, ports or packages if you do anything custom there, and anything else you might come up.
Exclusions will be necessary if you have files that should not be backed up. Maybe ~/.cache, but some files there under might be important, or not, depending. If you have ample disk space and cpu (and maybe bandwidth) it is easier to back everything up. Otherwise, time will need to be spent on an exclusion list and maybe also commands that clean out *.o compile files or other such temporary files before the backup is started.
Private keys are something else that might be good not to backup, especially if the backups go off elsewhere where who knows who could gain access to the backups, and maybe they can break the backup encryption? That might be bad. In this case you may need special backups for the private keys, or some means to regenerate or rotate them as need be, which may complicate installation of the system. So there are tradeoffs here.
One coworker once had wild card certificates and private keys for a domain under a git repository that was backed up unencrypted to all sorts of different places. Maybe this was not so good an idea, even if it did help automate the install of a new web host?
Virtual disk images for live virts are likely also problematic to backup and probably should be excluded. But I do not do much with those; usually the test virts are turned off.
Every so often one must practice a restore, maybe for real if you have a habit of deleting the wrong file, or support users who do so. Without such accidents, restores must be tested now and then. For one, this keeps you in practice with the software so things may go smoother when there is a critical emergency, and that the documentation can better be kept up to date. And another reason is that who knows if the backups are actually valid, without a test recovery every now and then?
$ doas mount /backup
$ doas restic -q -r /backup/restic restore -i ~/.kshrc -t /root/restore latest
...
$ doas find /root/restore -type f
/root/restore/home/jmates/.kshrc
$ doas umount /backup
If not the latest snapshot, then maybe try:
$ doas restic -q -r /backup/restic find --json somefile
Backup testing could be automated by placing known text into a known file on a known date, then restoring versions of that file to confirm the file contents are the expected value. But this might be too much effort and would require that the backup password be used in yet another place. A script to help automate the restore of a named file from a given date might however be useful, especially if that happens a lot.
It might also be helpful to practice a few system restores, though these can be time consuming. The ability to put an old disklabel onto a new disk might also be good to know, or at least to have notes somewhere on how to do it. An on-call sysadmin probably will want to practice for disasters (so they can better restore a system at 3 AM after not much sleep) more than an ordinary user would. Or whatever it is they are calling sysadmins these days.
In theory old backups will need to be removed to save space, or to comply with retention requirements, though my backup device is only seven percent full, and no backup is yet very old, so uh TODO.
doas restic -r /backup/restic forget --dry-run --keep-within=6m
And then run "prune" or replace the --dry-run option with --prune, in theory.
The man pages may not be complete so you may need to delve through other documentation. For example "forget" has flags for daily/weekly/monthly/yearly snapshots, but "backup" does not mention how to create such... maybe this is done via a tag?
tags #backup #restic