💾 Archived View for thebird.nl › gn-gemtext-threads › issues › systems › fallbacks-and-backups.gmi captured on 2022-07-16 at 15:49:10. Gemini links have been rewritten to link to archived content

View Raw

More Information

➡️ Next capture (2023-03-20)

-=-=-=-=-=-=-

Fallbacks and backups

As a hurricane is barreling towards our machine room in Memphis we are checking our fallbacks and backups for GeneNetwork. For years we have been making backups on Amazon - both S3 and a running virtual machine. The latter was expensive, so I replaced it with a bare metal server which earns itself (if it hadn't been down for months, but that is a different story).

Tags

• type: enhancement
• assigned: pjotrp
• keywords: systems, fallback, backup, deploy
• status: in progress
• priority: critical

Tasks

• [.] backup ratspub, r/shiny, bnw, covid19, hegp, pluto services
• [X] /etc /home/shepherd backups for Octopus
• [X] /etc /home/shepherd backups for P2
• [X] Get backups running again on fallback
• [ ] fix redis queue for P2 - needs to be on rabbit
• [ ] fix bacchus large backups
• [ ] backup octopus01:/lizardfs/backup-pangenome on bacchus

Backup and restore

We are using borg for backing up data. Borg is excellent at deduplication and compression of data and is pretty fast too. Incremental copies work with rsync - so that is fast. To restore the full MariaDB database from a local borg repo takes a few minutes:

wrk@epysode:/export/restore_tux01$ time borg extract -v /export2/backup/tux01/borg-tux01::BORG-TUX01-MARIADB-20210829-04:20-Sun
real    17m32.498s
user    8m49.877s
sys     4m25.934s

This all contrasts heavily with restoring 300GB from Amazon S3.

Next restore the GN2 home dir

root@epysode:/# borg extract export2/backup/tux01/borg-genenetwork::TUX01_BORG_GN2_HOME-20210830-04:00-Mon

Get backups running on fallback

Recently epysode was reinstated after hardware failure. I took the opportunity to reinstall the machine. The backups are described in the repo (genenetwork org members have access)

https://github.com/genenetwork/gn-services/blob/master/services/backups.org

As epysode was one of the main sheepdog messaging servers I need to reinstate:

• [X] scripts for sheepdog
• [X] enable trim
• [X] reinstate monitoring web services
• [X] reinstate daily backup from penguin2
• [X] CRON
• [X] make sure messaging works through redis
• [X] fix and propagate GN1 backup
• [X] fix and propagate IPFS and gitea backups
• [X] add GN1 backup
• [X] add IPFS backup
• [X] other backups
• [ ] email on fail

Tux01 is backed up now. Need to make sure it propagates to

• [X] P2
• [X] epysode
• [X] rabbit
• [X] Tux02
• [ ] bacchus