Remember how two months ago I started seeing sudden denial of service events on my tiny server. Load was creeping up to 20, 30, nearly 40. This server has two cores so the maximum should be 2. I started writing about it on 2024-09-18 Emacs Wiki and China. And I started blocking entire networks instead of just blocking individual IP numbers because I noticed that the global networks of hosting providers and the easy parallelization they offered meant that the same requests would come from many different IP numbers. By blocking the entire networks, I was blocking the IP number ranges of service providers that rented out their services to these maladapted programmers. If my goal is serving humans that browse the web, I don't feel bad about blocking network ranges that are used by machines.
2024-09-18 Emacs Wiki and China
Anyway, this little update is just to show that they're still at it. Over the weekend I noticed at one point how everything was slow. And it was getting slower. I managed to open a ssh connection to the server and noticed that load was up to 20 and climbing.
What had happened? I had rebooted the server on the evening of November 21st, late at night. Load started going up and on Sunday grow by another factor of two and everything was terrible. It's always on the weekend. Are they speculating on the admins being asleep at the wheel?
So, on the day of my marriage anniversary, I have to fight the fucking bots from China, once again.
I suspect that my ban list had not been restored correctly after the reboot. So what I did was this:
1. run ban-cidr in order to ban anybody the firewall forgot
2. identify some more bad actors using network-lookup
3. ban them by grepping the network-lookup output and piping it into a shell
4. add them to the ban-cidr file
Yay me! 🥳
But I wonder. Why do the Chinese bots love Emacs Wiki so much? It's the Emacs Wiki resources that shoot up when load shoots up.
And check out the WHOIS data I've been adding to the ban-cidr file. These requests originate all over China, from many different networks. It almost seems like a coordinated, national strategy. Are they looking for something?
Is the counter-revolution using Emacs??
​#Emacs ​#Administration ​#Butlerian Jihad
They're hitting on Emacs Wiki, again.
Memory usage of Emacs Wiki shooting up
Hypnotoad is trying to handle all the requests and started a lot of processes to handle Emacs Wiki.
Emacs Wiki processes going from 4 to 24
But now general system performance is suffering.
Number of threads going from 350 to 480
Network congestion leads to my fedi clients being unable to refresh. Now it's getting personal.
TCP connections going from about 100 per second up to 900 per second
I'm going to keep adding to `ban-cidr` but in order to save time I'm going to skip whois. Take a look at network-lookup-lean.
I added 1992 networks to the `ban-cidr` script, just now.
I manually did some checks and it's not China any more! The last entry I added is from Jordan, the one before that from Brazil, then a few from Urugay, more Brazil, one USA.
I kept on blocking them. Now up to 2831 networks.
Networks banned today: 5404.
We're at 8901 banned networks, today.
Added another 1866.
Here's how I'm doing it. First, no more whois lookups. Sad, but necessary!
# rcidonly is a sign that this is a bot tail -n 5000 /var/log/apache2/access.log \ | grep -v ^social \ | grep "rcidonly" \ | bin/admin/network-lookup-lean > result.log # Check how many there are grep ipset result.log|wc -l # Block them grep ipset result.log|sh # Document the block grep ipset result.log>>bin/admin/ban-cidr
But they're still at it! Ten thousand blocks later.
Continued here: 2025-01-23 The bots are at it again.
