Over the past few days, I've been battling a pernicious bug in my gopher server [1] wherein it becomes CPU (Central Processing Unit) bound and cause other issues on the server. I then have to go in and kill the gopher process (and the one time I couldn't even do that—I had to have the virtual server restarted). I initially attributed this to an over-aggressive bot crawling my site and blocked it with iptables but even that didn't solve the issue.
The problem is—nothing to my knowledge has changed on my virtual server, nor the server that it is running under, nor the network it's on. My Gemini server [2] gets way more traffic than my gopher site [3] and it's fine, and the only difference between the two is—the Gemini server [4] uses TLS (Transport Layer Security) but otherwise, is nearly identical to the gopher server.
It's very odd.
The other day I added some code (in a branch, not in the main line version) to log memory usage, number of threads (technically, Lua coroutines), number of running threads, number of waiting threads, and number of active sockets. And since adding that, the gopher server has been running fine, but just now I do see a potential problem—the number of threads is two higher than the number of actual connections, which “shouldn't” happen.
Woot! I now have a lead on the problem!
But I do wonder what recently caused the issue? The code hasn't changed since April, and now I'm wondering if my Gemini server has a similar issue, since the code bases are similar in nature.
[1] https://github.com/spc476/port70
[2] gemini://gemini.conman.org/
[3] gopher://gopher.conman.org:70/1