Recently I have noticed that I’m sometimes banned from my own websites. That is, the site is not reachable, but when I check any of the “is the site down for everybody or is it just me?” sites, it’s always just me. I also cannot SSH to the machine unless I use the IPv6 address directly.
OK, so fail2ban is banning the IPv4 of my home network. Why? Is some app I’m using bombarding the site with requests? Let’s check.
Visiting ip4.me tells me my IPv4. Grepping `/var/log/fail2ban.log` I see that it has been banned at 11:03 and 13:03 today, by the `alex-apache` rule.
This rules counts *every access* as a potential fail and ignores some URLs that I deem to be harmless (static files such as CSS files, fonts, pictures, podcast episodes, PDF files, and so on:
failregex = ^(www\.)?(alexschroeder\.ch|arabisch-lernen\.org|campaignwiki\.org|communitywiki\.org|emacswiki\.org|flying-carpet\.ch|korero\.org|oddmuse\.org|orientalisch\.info):[0-9]+ <HOST> ignoreregex = ^[^"]*"GET /(robots\.txt |favicon\.ico |[^/ ]+.(css|js) |cgit/|css/|fonts/|pics/|export/|podcast/|1pdc/|gallery/|static/|munin/|osr/|indie/|rpg/|face/|traveller/|hex-describe/|text-mapper/|contrib/pics/)
This is from `/etc/fail2ban/filter.d/alex-apache.conf`. The actual limits are defined in `/etc/fail2ban/jail.d/alex.conf`:
[alex-apache] enabled = true port = http,https logpath = %(apache_access_log)s findtime = 40 maxretry = 20
So basically you’re allowed 20 requests in 40s, not counting the requests matching `ignoreregex`.
OK, let’s see the requests. Here’s a little Perl script I wrote:
#!/usr/bin/env perl while (<STDIN>) { m/^(\S+:\d+) ([0-9.]+) - - \[(.*?)\] "(.*?)" (\d+) (\d+|-) "(.*?)" "(.*?)"/ or warn "Cannot parse:\n$_" and next; my ($host, $ip4, $date, $request, $code, $size, $referrer, $agent) = ($1, $2, $3, $4, $5, $6, $7, $8); $requests{$request}++; $total++; } @result = sort {$requests{$b} <=> $requests{$a}} keys %requests; foreach $label (@result) { printf "%70s %10d %3d%%\n", $label, $requests{$label}, 100* $requests{$label} / $total; }
Let’s use it to report on today’s log entries from my IP:
GET /pdfs/spellcasters/ HTTP/1.1 12 19% GET /pdfs/ HTTP/1.1 12 19% GET /jewelry HTTP/1.1 8 13% GET /wiki/Apps HTTP/1.1 8 13%
Hm, who the hell is requesting `/pdfs/spellcasters/` and `/pdfs/`‽ Not me, that’s for sure! At least not that I know of. Then again, I’d say that the `/pdfs` directory contains just static files so adding that to the `ignoreregex` should be no problem.
I still wonder who does this, though. Is it the Firefox app on iOS?
#Administration #fail2ban
(Please contact me if you want to remove your comment.)
⁂
I recently read an argument against using fail2ban. It basically went like this:
1. if your SSH is set up correctly, nobody is getting in
2. if you change your default port, you will waste hours of your life searching for the correct rsync and scp flags every single time you use them (true!)
3. if you run fail2ban, the above is still true
– Alex Schroeder 2020-06-14 20:50 UTC
---
@deshipu commented on a recent incident:
I like to trace the route in such situations to at least see at which point along the way the problem is. – 2021-07-30 The web server locking me out
2021-07-30 The web server locking me out
Must remember that.
– Alex 2021-07-30 09:26 UTC
---
We went to a hotel, and then we connected all our devices to the hotel wifi. Often, I was banned, that is to say, the hotel’s router was banned. At first, I was able to connect to the server using mosh but then that stopped working as well.
I had to use a different SSH host (thank you, zeus!) as a jump host. Here’s how to connect to some other host and forward localhost port 2222 to my server’s port 882:
ssh shire -L 2222:alexschroeder.ch:882
And in a second shell:
ssh -p 2222 localhost
This allowed me to log into my own host even though my IP was banned and logins are disabled. Lucky!
But why was the hotel’s IP getting banned? The answer is in the logs. Grepping for the hotel’s IP number got me a ton of lines like the following:
alexschroeder.ch:443 … - - [31/May/2022:22:20:32 +0200] "OPTIONS /radicale/alex/ HTTP/1.1" 200 6143 "-" "iOS/15.4.1 (19E258) remindd/1.0" alexschroeder.ch:443 … - - [31/May/2022:22:20:33 +0200] "PROPFIND /radicale/alex/ HTTP/1.1" 401 6204 "-" "iOS/15.4.1 (19E258) remindd/1.0" alexschroeder.ch:443 … - - [31/May/2022:22:20:33 +0200] "PROPFIND /radicale/alex/ HTTP/1.1" 207 7373 "-" "iOS/15.4.1 (19E258) remindd/1.0"
It’s radicale! All the devices are trying to sync their calendars at the same time, with a ton of WebDAV requests (OPTIONS and PROPFIND instead of GET or POST) – and these requests are handled, forwarded and logged by Apache, and fail2ban monitors the Apache logs, so they all count!
The solution was to add something to the “ignoreregex” in “/etc/fail2ban/filter.d/alex-apache.conf”:
# Author: Alex Schroeder <alex@gnu.org> [Definition] # Most sites in the logfile count! What doesn't count is fedi.alexschroeder.ch, or chat.campaignwiki.org. # communitywiki.org:443 000.000.000.000 - - [24/Aug/2018:16:59:55 +0200] "GET /wiki/BannedHosts HTTP/1.1" 200 7180 "https://communitywiki.org/wiki/BannedHosts" "Pcore-HTTP/v0.44.0" failregex = ^(www\.)?(alexschroeder\.ch|arabisch-lernen\.org|campaignwiki\.org|communitywiki\.org|emacswiki\.org|flying-carpet\.ch|korero\.org|oddmuse\.org|orientalisch\.info):[0-9]+ <HOST> # Except cgit, css files, images... # alexschroeder.ch:443 0:0:0:0:0:0:0:0 - - [28/Aug/2018:09:14:39 +0200] "GET /cgit/bitlbee-mastodon/objects/9b/ff0c237ace5569aa348f6b12b3c2f95e07fd0d HTTP/1.1" 200 3308 "-" "git/2.18.0" ignoreregex = ^[^"]*"(GET /(robots\.txt |favicon\.ico |[^/ ]+.(css|js) |cgit/|css/|fonts/|pdfs/|txt/|pics/|export/|podcast/|1pdc/|gallery/|static/|munin/|osr/|indie/|rpg/|face/|traveller/|hex-describe/|text-mapper/|contrib/pics/)|(OPTIONS|PROPFIND) /radicale)
– Alex 2022-05-31 20:25 UTC