💾 Archived View for jsreed5.org › log › 2022 › 202202 › 20220213-hypervisor-troubles.gmi captured on 2022-06-11 at 21:14:58. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2022-03-01)
-=-=-=-=-=-=-
---
The headless git repository that acts as the "master" repo for Rob's Capsule is hosted on a virtual machine powered by Qemu. I also host a few other services, most notably a private Minecraft server for my friends.
On Thursday night I brought the Minecraft server down to apply some OS updates, and while I was at it I decided to apply updates to the hypervisor. The process was agonizingly slow: it took over an hour to download all the required packages, and the update itself had to run overnight. That should have been a red flag to me, but I didn't bat an eyelash when I ran a restart command afterward.
Before leaving for work on Friday morning, I tried to log into the hypervisor to check on it, but I couldn't connect over SSH. Connecting a crash cart revealed that the machine had failed to mount /sysroot. Further errors indicated that several sectors on the OS HDD could not be read. My boot drive was dead. I didn't have sufficient time to troubleshoot the problem before leaving for work, so for the rest of the day I was unable to SSH into my home network or interact with my capsule.
Fortunately, when I built my home server, I had the sense to put all the VMs onto a physically separate SSD. Though the boot drive was dead, all the VMs were unharmed, and they could run just fine once I got the hypervisor up and running.
Several hiccups occurred along the way--more than there should have been, considering my level of professional experience--but to make a long story short, the VMs finally came back online early this afternoon. I can now post on my capsule again.
Now I just need to fix a lingering issue with my twtxt file. When you're a system administrator, the troubles never seem to end.
---
[Last updated: 2022-02-13]