Sorry if you've signed up for email notification and gotten some wierd notifications over the past twelve hours. That's because there was a show-stopping bug in the code that only now was triggered.
You see, I rewrote the email notification code some time ago, but hadn't fully tested it yet. Then, a few days ago I decided to add in custom error pages and kept having issues. I knew I had done it before, then I realized I did, over there [1], but the custom error pages weren't working over here [2]. I thought maybe it was a different version of the code.
So I checked out a fresh copy over at the other place [3] to see if I broke whatever it was that made the custom pages work (it was using an older code base there), and it still worked. But since I'm not using email notification over there, the new email notification code was still untested.
So I checked out a fresh copy here [4]. The custom error pages were still broken.
Turns out it was the installation of Apache [5] on the server here (I used the distro-installed version for a change)—seems that it has a directory alias for /error/, which just so happens is what I named the directory. Had I named it /errors/ (plural) things would have worked the first time and all of this wouldn't have happened (or rather, would have happened at a later time).
Sigh.
But I did.
But notice that I hadn't tested the email notification code yet.
Until yesterday and today.
And boy, did things blow up.
The program kept crashing. Normally, that isn't much of an issue (well, okay, it is an issue) because normally, one is in front of the program to watch it crash, but in this case, I wasn't aware of the crash, because I used the email interface.
Now, mod_blog takes the entry (from a web page, a file, or through email), adds it to the archive, generates the new page, and then, if enabled, sends the email notifications.
It was crashing (unbeknowst to me) during the email notifications.
Postfix [6] has been instructed to send emails to a particular address to mod_blog, which it does. But Postfix also notices that the program crashed. So it requeues the email.
And when the program crashes, the update lock it has is released (technically, it's a file lock that the kernel maintains, but hey, the program goes away, so does the lock).
Next entry comes along, process repeats, and Postfix realizes it has some email queued up, and tries to send that at the same time. Now we have two processes trying to update the blog. Normally, this isn't an issue because of the update lock. But the lock is released in an uncontrolled state and …
Things blow up and get ugly.
But what I'm seeing is a blog that failed a validation check, I fix it, only to have it fail the validation check again, which I fix, only to fail the validation yet again, only now I notice that the same entry I thought I fixed twice keeps coming back to haunt me and even worse, the entire front page is a mess, and is remaining a mess no matter what I do.
I tracked the problem down, but I may have sent some spurious notifications in doing so. Once I realized it was the new email notification code (and man, how did I ever check that code in? Leaks memory like you wouldn't believe! Sheesh) I reverted to the old version, cleared out the mail queue and hopefully, got this situation under control.
[1] http://www.saltminechronicles.com/foo
[2] https://boston.conman.org/
[3] http://www.saltminechronicles.com/