I was checking up on the Apache error log [1] when I noticed the following:
[Sun Jul 28 18:47:21.455848 2024] [authz_core:error] [pid 25597:tid 3916331952] [client 74.173.118.3:53916] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known [Sun Jul 28 18:47:59.176743 2024] [authz_core:error] [pid 25598:tid 3916331952] [client 74.173.118.3:53918] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known/ [Sun Jul 28 18:50:33.324290 2024] [authz_core:error] [pid 25759:tid 3832404912] [client 74.173.118.3:53922] AH01630: client denied by server configuration: /usr/local/apache2/htdocs/.well-known
That's odd, I thought. I don't have that directory in any of my virtual domains, so why is it denied by the server configuration? And thus I fell into a rather odd rabit hole of Apache configuration oddities.
I created the directory. I can see it when I go to https://boston.conman.org/.well-known/. But when I go to http://boston.conman.org/.well-known/ I would get a “403 Forbidden” error, and the above error message logged. The only difference between the two links—one is HTTPS (HyperText Transfer Protocol Secure) (that works) and the other is HTTP (HyperText Transfer Protocol) (that fails). But if I go to http://boston.conman.org/ (HTTP—thus insecure), it would redirect to https://boston.conman.org/ (HTTPS—secure). In fact, every link to boston.conman.org via HTTP redirects, except for those starting with /.well-known/.
Huh?
It turns out, this started a a year and a half ago [2] when I enabled the Apache [3] module mod_md [4] and used the MDRequireHttps directive. This directive will cause plain HTTP requests to be redirected to HTTPS, because I know, I just know, that one day Google is going to take HTTP out behind the shed and then no one will be able to use plain HTTP anymore because The All Knowing Google knows whats best for us (All Praise Google, Peace Be Upon It!) so I might as well get in front of that before it happens. But there's a small bit in the documentation:
## MDRequireHttps Directive
…
You can achieve the same with mod_alias [5] and some Redirect [6] configuration, basically. If you do it yourself, please make sure to exclude the paths /.well-known/* from your redirection, otherwise mod_md might have trouble signing on new certificates [emphasis added].
Okay, so that explains why http://boston.conman.org/.well-known/ (HTTP—insecure) isn't being redirected—it's a side effect from mod_md. But that doesn't explain the error where it's denied by the server configuration. A bit more digging, and I find the following in the Apache configuration:
DocumentRoot "/usr/local/apache2/htdocs"
<Directory "/usr/local/apache2/htdocs">
#
# Possible values for the Options directive are "None", "All",
# or any combination of:
# Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important. Please see
# http://httpd.apache.org/docs/2.4/mod/core.html#options
# for more information.
#
Options Indexes FollowSymLinks
#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
# AllowOverride FileInfo AuthConfig Limit
#
AllowOverride None
#
# Controls who can get stuff from this server.
#
#Require all granted
Require all denied
</Directory>
Aha! When I set things up, I configued the HTTP site with:
<VirtualHost 71.19.142.20:80> ServerName boston.conman.org Protocols h2 h2c http/1.1 acme-tls/1 </VirtualHost>
I don't with a directory because I know that all requests will be redirected anyway, so why bother? Only in this case, mod_md isn't doing a redirect for /.well-known/ and because there's no <Directory> directive, the requests fall back to the Apache default web directory, which is configured to be unreadable for every request.
Wow!
A quick change to make the default web directory available and no more AH01630 errors. So now I wait to see if this breaks anything. Wheeeee!
[4] https://httpd.apache.org/docs/2.4/mod/mod_md.html
[5] https://httpd.apache.org/docs/2.4/mod/mod_alias.html
[6] https://httpd.apache.org/docs/2.4/mod/mod_alias.html#redirect