💾 Archived View for thebird.nl › gn-gemtext-threads › issues › systems › tux02-production.gmi captured on 2023-04-19 at 23:25:55. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2023-01-29)

-=-=-=-=-=-=-

Tux02 Production

We are going to move production to tux02 - tux01 will be the staging machine. This machine is aimed to be rock solid. The idea is to have 4-6 times a year upgrades. Also we should be able to roll back on an upgrade and be able to create time machines.

Tags

• assigned: pjotrp
• status: in progress
• priority: medium
• type: system administration
• keywords: systems, tux02, production

Tasks

• [X] update guix guix-1.3.0-9.f743f20
• [X] set up nginx (Debian)
• [X] test ipmi console (172.23.30.40)
• [X] test ports (nginx)
• [?] set up network for external tux02e.uthsc.edu (128.169.4.52)
• [X] set up deployment evironment
• [X] sheepdog copy database backup from tux01 on a daily basis using ibackup user
• [X] same for GN2 production environment
• [X] sheepdog borg borg the backups
• [X] start GN2 production services
• [X] add GN3 aliases server
• [X] add Genenetwork3 service

env FLASK_APP="main.py" flask run --port=8087

• [X] add proxy
• [X] set up databases
• [X] set up https and letsencrypt
• [X] set up firewalling
• [X] set up systemd
• [ ] setup logrotate for production log files
• [ ] run git automatically on /etc and backup without passwords
• [ ] add borg backups
• [ ] create check list for manual testing
• [ ] look at performance

Info

We have a protocol for updating GN2 on Tux02.

Restore database from backup

Databases no longer get copied. We only restore from backup. First because these are reproducible [installs]. Second because the backup should be in a sane state(!).

Restoring a database from backup (about an hour)

root@tux02:/export3/backup/tux01/borg# borg extract borg-tux01::borg-backup-mariadb-20211024-03:09-Sun --progress

Next move the dir to fast storage.

Symlink /var/lib/mysql

The database is symlinked. You can point that to the recovered backup. Restart the DB and run mysql_upgrade followed by our tests. E.g.

systemctl stop mariadb
ln -s /export2/mysql/borg-backup-mariadb-20211024-03\:09-Sun /var/lib/mysql
systemctl start mariadb
/usr/local/guix-profiles/gn-latest-20211021/bin/mysql_upgrade -u webqtlout -pwebqtlout
/export/backup/scripts/tux02/system_check.sh