So, what was the ofending program? [1]
Glad you asked (whince).
It turned out to be mod_blog [2], the program that runs this very site.
A friend of mine (who for now wishes to remain anonymous) is interested in blogging and I said I would set him up with my system. So I create a site for him, copy over my existing template, modify the configuration for him, etc., etc. I then send in (via email) the first post to see if things work.
Well, that's where things didn't work.
The post itself was accepted and stored correctly.
Small digression: The Boston Diaries is primarily dynamic. You type in something like http://boston.conman.org/2002/3/4 [3] and that page is generated on the fly. I change the template, the effect takes place immediately. However, the main page, the one you get by going to http://boston.conman.org/ [4] is not dyamic—it's actually a static page recreated whenever a new entry is posted. It doesn't have to be, but I figure that since this page is probably going to be loaded most often I might as well cache a static copy to keep the system load down.
So, part of the process of accepting and storing an entry is the generation of the main page. Normally, it works fine. But not in this case.
Another small digression: the configuration file for mod_blog needs the starting date of the blog. There are cases where I need this information and instead of wasting a lot of time going backwards from now finding the first entry, again, it's cached information.
I had thought that I may have made a mistake in the starting date. No, I got the starting date correct. What I didn't get correct was handling the situation when a blog is actually started.
When writing the software, I had already been keeping entries. In fact, I think I had a month or so [5] worth of entries when I started the code two years ago. And I was so focused on getting the URL (Uniform Resource Locator) processing correct, that I neglected to test some border cases. And I never got around to testing those cases since they didn't affect me.
Until now.
Oops.
Problems I found:
Cases I should have handled (and tested for!) but neglected.
I do need to really go through the code and clean it up.
The problem I had was that I'm a bit too close to the code. I'm working towards a specific goal (new method of document storage retrieval and reference) and as such, the software is experimental and the problems I'm focusing on meant I missed some reliability details elsewhere, since hey, it works in my case.
And while some people have probably grabbed the software I doubt many, if any, are actually using the software since I'm not getting any feedback on the code itself. Okay, you do have to hunt around to find the link to the source code [7] but it has been downloaded. And I'm sure it being written in C makes it all that much more popular. [8]
But little did I expect software I wrote to crash a Unix server. The last time I saw userland software (an application) crash a Unix server was … oh … eight years ago [9] I think.
[2] https://boston.conman.org/about/
[4] https://boston.conman.org/
[6] https://boston.conman.org/bostondiaries.rss