💾 Archived View for thebird.nl › gn-gemtext-threads › topics › systems › restore-backups.gmi captured on 2023-06-14 at 14:32:10. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Restore backup

Borg

We use borg backups. Load a recent borg, for example from my profile

source /home/wrk/opt/borg/etc/profile

The backup passphrase needs to be set.

source /home/ibackup/.borg-pass

Now run against a repo on 'tux02'

borg list /export3/backup/tux01/borg/borg-tux01/
borg-backup-mariadb-20221004-03:13-Tue Mon, 2022-10-03 22:13:11

The last backup on 'tux02' is from October 2022 - after I did a reinstall. That is not great.

According to sheepdog the drops are happening to 'space' and 'epysode', but 'tux02' is missing:

https://rabbit.genenetwork.org/sheepdog/index.html

Mariadb

The first step is to find where the current mariadb is running. This will be a symlink in /var/lib/mysql -> some dir.

mysql -> /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun

We typically run the database on an nvme partition. Check if there is enough space(!). It may be you need to remove the old database after making a backup (if it is not in borg).

Next, find the backup. Typically it is symlinked from /export/backup. On

tux02:/export5/backup/bacchus/drop/tux01

Unpack from backup

We need a place to unpack the DB. On tux02 export5 is large, so let's try that and pick the last backup

cd /export5/mysql
tux02:/export5/mysql# borg list /export5/backup/bacchus/drop/tux01/borg-tux01
borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230207-02:31-Tue

As it happened, this was an incomplete backup! Check original size with

borg info --last 3 /export5/backup/bacchus/drop/tux01/borg-tux01

so I went with

borg extract --progress /export5/backup/bacchus/drop/tux01/borg-tux01::borg-backup-mariadb-20230205-03:38-Sun

Mariadb's own ibdata1 is still at 100Gb. Too large, and we'll muck it out at some point. The total mariadb database at this point is 430Gb.

Move DB in place

To move the new DB in place we first have to stop mariadb, move the old database out of the way, move the new one in, and restart the DB after making sure permissions and symlink are right.

Just to be sure we copy the old DB somewhere.

systemctl stop mariadb
cd /export2/mysql
cp -vau /export2/mysql/borg-backup-mariadb-20211024-03:09-Sun .
root@tux02:/export2/mysql# cp -vau /export5/mysql/home/backup/tux01_mariadb_new/latest/* borg-backup-mariadb-20230205-03:38-Sun
cd /var/lib
rm mysql
ln -s /export2/mysql/borg-backup-mariadb-20230205-03\:38-Sun mysql
systemctl start mariadb
systemctl status mariadb

Oops. Can't create test file /var/lib/mysql. We need to set permissions.

root@tux02:/var/lib# chown mysql.mysql mysql

And we are running.

Next stop update genotype files.

Files

GN has a file repo for genotype files, lmdb and such, next to MariaDB. Typically

root@tux02:/home/gn2/production/genotype_files# du -sh .
38G     .

the backup in drop lists, for example

root@tux02:/export5/backup/bacchus/drop/tux01# borg list borg-genenetwork/
borg-GN2-home-20230207-04:00-Tue     Mon, 2023-02-06 22:00:02
borg-GN1-home-20230207-04:40-Tue     Mon, 2023-02-06 22:40:01
borg-GN3-home-20230207-04:41-Tue     Mon, 2023-02-06 22:41:27
borg-GN2s-home-20230207-04:41-Tue    Mon, 2023-02-06 22:41:45
borg-ZACH-home-20230207-04:42-Tue    Mon, 2023-02-06 22:42:01
borg-datafiles-home-20230207-04:42-Tue Mon, 2023-02-06 22:42:32

At this poing ZACH-home is the main dir, so this fetches all files

borg extract --verbose borg-genenetwork::borg-ZACH-home-20230207-04:42-Tue home/zas1024/gn2-zach/genotype_files

and we copy it to /home/gn2/production.

We also have a separate data files 'warehouse', but that is not linked up with the webserver right now.