💾 Archived View for bbs.geminispace.org › u › jsreed5 › 16908 captured on 2024-08-31 at 13:56:59. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-08-18)
-=-=-=-=-=-=-
Re: "My intermittent capsule outages are being caused by what..."
Good point! The crawler's IP address is 104.207.150.107.
May 13 · 4 months ago
Reverse DNS resolves to celery.eu.org; over HTTP it says 'unplanned maintenance', copy-pasting the IP into the browser redirects you to a rickroll. TBH I would just keep the domain in your blacklist for now.
My intermittent capsule outages are being caused by what appears to be a very aggressive crawler. The capsule's robots.txt file tells bots not to index my CGI scripts, but this crawler is ignoring the file and sending multiple requests per second against my scripts, which overloads the server and causes it to crash. I've temporarily solved the problem by blocking the crawler entirely; I'll look for a more permanent solution.