💾 Archived View for thebird.nl › gn-gemtext-threads › topics › systems › backup-drops.gmi captured on 2023-06-14 at 14:32:09. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Backup Drops

To make backups we use a combination of sheepdog, borg, sshfs, rsync. sheepdog is the monitor and status is tracked using a redis queue. borg does incremental backups as a local user. Next we drop the backup to a remote machine using a special user on the remote with very limited access - only one directory can be mounted with sshfs and there is no ssh login. Finally the data gets rsync'd across.

This system proves pretty resilient over time. Only on the synology server I can't get it to work because of some CRON permission issue.

Info

Borg backups

It is advised to use a backup password and not store that on the remote.

Running sheepdog on rabbit

sheepdog monitor running on redis queue

current web monitor for GN sheepdog on rabbit

Create a drop user on remote

adduser bacchus

Disable the password, just to be sure. Next add the following to /etc/ssh/sshd.conf:

-# override default of no subsystems
-Subsystem      sftp    /usr/lib/openssh/sftp-server
+Subsystem      sftp    internal-sftp -f AUTH -l VERBOSE
+
+Match User bacchus
+    # ChrootDirectory /export/backup/%u
+    ChrootDirectory /export/backup/%u
+    X11Forwarding no
+    AllowTcpForwarding no
+    ForceCommand internal-sftp
+    PasswordAuthentication no
+    PubkeyAuthentication yes

And run

systemctl reload ssh

You may need to add to allowusers for ssh access. If you use allowusers (recommended) you can even specify the sending host with

AllowUsers bacchus@remote

where remote can be an IP address.

Warning: if you introduce this `AllowUsers` command all users should be listed or people may get locked out of the machine.

Next create a special key on the backup machine's ibackup user (just hit enter):

su ibackup
ssh-keygen -t ecdsa -f $HOME/.ssh/id_ecdsa_backup

and copy the public key into the remote /home/bacchus/.ssh/authorized_keys

Now test it from the backup server with

ssh -v bacchus@dropserver

it should give a Permission denied (publickey).

On the drop server you can track messages by

n0tail -40 /var/log/auth.log

ssh -v -i ~/.ssh/id_ecdsa_backup bacchus@dropserver

should give a Broken pipe(!). In auth.log you may see something like

fatal: bad ownership or modes for chroot directory component "/export/backup/"

This is a tricky bit. This directory should be owned by root and have permissions. The inside user directory has different permissions:

ls -ld /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:08 /export/backup/
drwxr-xr-x 3 root root 4096 Oct 21 02:07 /export/backup/bacchus
drwx------ 3 bacchus bacchus 4096 Oct 21 02:26 /export/backup/bacchus/drop

So, as root

cd /export
mkdir -p backup/bacchus/drop
chown bacchus.bacchus backup/bacchus/drop/
chmod 0700 backup/bacchus/drop/

If auth.log says error: /dev/pts/11: No such file or directory on ssh, or received disconnect (...) disconnected by user we are good to go!

Next try sshfs

su ibackup
mkdir -p ~/mnt/dropserver
sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/dropserver/
df -h|grep mnt
touch ~/mnt/dropserver/drop/HELLO

And the remote directory should be ready for dropping files!

To unmount the dir

fusermount -u ~/mnt/dropserver

IMPORTANT: it is important to try ssh and read /var/log/auth.log to deal with permission issues. sshfs and the underlying sftp protocol are fussy.

Using rsync over sshfs with sheepdog

A backup script with sheepdog may look like

sheepdog_run.rb -h rabbit --always -v --tag "drop-mount-dropserver" -c "sshfs -o IdentityFile=~/.ssh/id_ecdsa_backup bacchus@dropserver:/ ~/mnt/this"

sheepdog_run.rb -h rabbit --always -v --tag "drop-rsync-dropserver" -c "rsync -vrltDP this/* borg/* ~/mnt/this/drop/this/ --delete"

sheepdog_run.rb -h rabbit --always -v --tag "drop-unmount-dropserver" -c "fusermount -u ~/mnt/this"

It may be useful to add the following options to sshfs:

sshfs -o reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,IdentityFile=~/.ssh/id_ecdsa_backup ...

The recent scripts can be found at