When I try to log into the server using SSH it takes a long time. Why is that? `iotop` doesn't show anything suspicious. And yet, since Saturday 18:00, `munin` shows that something is up.
System CPU goes up. Everything goes up!
Throughput per device starts to be weird.
Utilisation per device goes up to 50%.
I rebooted the server. "Have you tried turning it off and on again?" 🤷
​#Administration
And that seems to have worked. I wonder what I did back then.
Saturday 18:00 shows users connecting (me). Doing something. Who knows what.
I was clearly doing something!
Here's what I noticed: I uploaded a new version of the Norn bot, which had an error. I replaced it. Was systemd spawning it again and again, spamming the journal? Is that what caused the problem?
This is very strange.
But… rebooting helped!
CPU is back to normal, lots of idle time.
Disk utilization is back to normal, nearly 0.
Another spike from 0:00 to 2:00.
Strangely enough, this spike in CPU usage was not accompanied in a spike in disk usage.
Another thing I noticed this morning: Not only is my SSH extremely sluggish – sometimes the command line takes half a minute or longer to update – but my IPv4 banlist also wasn't around. I had to run ban-cidr again to reconstruct it. I've been banning some more networks, too (mostly Contabo and Hetzner in Germany). It doesn't seem to be helping right now.
And while I'm snooping around: This hourly beat in Apache requests seems like a clear sign of bot activity, too. Fuckers.
Every hour there is a dip. it looks like an inverse heartbeat.
