💾 Archived View for m0yng.uk › 2021 › 02 › Autopsy---Breaking-a-server captured on 2021-12-04 at 18:04:22. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-03)
-=-=-=-=-=-=-
Created 2021-02-25
Ok, I have to hold my hands up here. I screwed up. I broke the mastodon.radio server and I shouldn't have.
I'm sorry.
Here I'll try to walk through what happened, and how I fixed it.
I thought it would be a good idea to add nitter[1] as a service to mastodon.radio, I thought as people using the server are actively NOT using twitter they might like a proxy to view the odd tweet, and the pubic nitter servers are very often overloaded.
1: https://github.com/zedeus/nitter
This wasn't a mistake, but it did quickly lead to one, that then led to another, and then a cascading failure leading me to a fairly bad place.
Nitter needs nim[2] - but it needs version >= 1.2, which isn't available in debian stable (at time of writing that's 0.19.4). Nor is it in the (relatively safe) backports repo (at time of writing that's 1.0.4). But testing has 1.4.0! So knowing that I can selectively install just one package from backports I thought I could do the same from testing.
Now would have been a good point to stop, and say it's not worth it. Or try docker, or something else.
I added testing, and apt said that nearly everything needed updating. Which wasn't a shock. But wasn't what I planned.
I thought I would try "pinning" everything, debian has a concept of priorities where different packages can be sourced from, so I set up some preferences which put the testing files as a low priority.
The file looks a little like this
Package: * Pin: release a=testing Pin-Priority: 900
This kinda worked, apt no longer wanted to update everything.
So, I tentatively try running apt install again, this time asking to get nim from testing. This listed a bunch of dependencies, which I thought was probably too long but I didn't read it in full.
Now would have been a good point to stop. At this point I've broken it, I just don't know it yet.
I told apt to go ahead and install stuff. And it did.
Then I noticed that my connection to the xmpp server had gone, and I was getting lots of authentication failures in the logs. I restarted the server and the auth script (which uses python) failed. I tried running it manually and it was missing a library. So I tried installing it with pip and that failed too.
I think it was this point I decided to try backing out of this.
If I remember correctly I tried to uninstall nim at this point, and then I tried to make apt downgrade things back to stable. This failed, and I got an ominous message amongst the failure.
error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
So I tried again, but sudo said no
sudo: account validation failure, is your account locked?
I tried to ssh in again
kex_exchange_identification: read: Connection reset by peer
At this point I still had an SSH connection, but couldn't sudo, couldn't create a new connection, couldn't su, basically I couldn't do ANYTHING.
I could ssh OUT so I began backing up some data to another server (this one!)
I also made use of the VPS snapshot facility, and created a snapshot of the broken server.
I sent an email to mythic-beasts who host the VPS and they were both helpful and a little pessimistic
that's probably quite tricky to recover from.
They suggested booting a live cd (virtual, of course) and I took this advice!
I booted systemrescuecd and connect via their VNC thing, which let me mount the filesystem. I used wget to download the libc6 deb which contains libcrypt.so.1 and manually unwrapped the file, and extracted the data over the server's filesystem.
I crossed my fingers and rebooted.
It worked! I was able to log in! (still using VNC, ssh wasn't running) and tried running apt again, which gave me a long list of broken packages. It suggested running apt --fixall, so I did. But it didn't work, saying that apt-listchanges was at fault. So I tried installing it again, but apt was still not happy and said to run fixall. Anything I did with apt said the same. apt uses python, and debian 10 expects python3.7, which it seemed I didn't have anymore. I couldn't install anything using apt, so I started downloading the packages manually and using dpkg to install them, which has the advantage of telling me about dependencies that are missing.
I don't know what went wrong, but at some point things stopped working. Almost every command triggered a "stack smashing detected" error and terminated, even ls!
So, I started restoring the snapshot.
Then I had dinner.
Once the snapshot was restored, I booted up systemrescuecd again and this time tried to go straight into a chroot of the server, which didn't work because I didn't have libcrypt.so.1
So I backed out of that, extracted the file deb again, and tried the chroot again. This time apt worked! But still had many issues. So I carefully ran through the dependency tree of python3 on debian, and worked from the bottom up downloading the deb files (oh, did I mention I couldn't paste? So I was manually typing the URLs) and using dpkg to install them until I had python3 available as a command.
I then ran apt --fixall again, and it worked!
I then quickly removed the testing repo, removed the pinning, and ran apt update, then apt upgrade, then apt dist-upgrade, and then apt autoremove, until nothing else was complaining.
And I crossed my fingers, and rebooted.
Some things still weren't working, but at least the server was running and mastodon had (mostly) started.
I had not removed the pinning for stable completely, and nodejs on debian is v10 and mastodon-streaming needs v12 so it wasn't starting. I removed the stable pin completely and apt installed v12. Huzzah! mastodon-streaming started!
Seen as this server is mastodon.radio I thought at this point we were ok, and I broke away to write this post.
ejabberd is still complaining, but it's not mission critical and I can fix it later.
-+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+-
🖤 Black Lives Matter
💙🤍💜 Trans Rights are Human Rights
❤️🧡💛💚💙💜 Love is Love
Copyright © 2021 Christopher M0YNG
Code snippets are licenced under the Hippocratic License 3.0 (or later.)
Page generated 2021-11-02 by Complex 19