Well, that was fast. I installed the new version of mod_blog [1], went out to dinner with Bunny, and came home to over 80 crash reports, reason: “Address not mapped for object.” Or in other words, “the program tried to access memory that didn't belong to it.”
Bad news—it wasn't every request, just certain ones. Even worse news—the logging server and the web server have a 3′50″ difference in time (that took me about fifteen minutes to realize). I also realized that even with stack dump and trace, there still wasn't enough contextual information to figure out what was going on.
I decided to dump the command line arguments (a hack—I have to store the arguments in a global variable and make crashreport(), a library function, depend upon these global variables, which is bad form) and the environment variables (less of a hack—the system already has these in a global variable). With that, I was able to track down the issue.
The issue—some script kiddies are trying to hit the webpage that creates webpages. This isn't normally an issue, since the POST method requires authentication, but these script kiddies were trying to create entries via GET, and the error that was generating wasn't being properly checked (since that shouldn't happen—oops). And because of that, some unititialized variables were dereferenced and boom! A crash report.
The surprising thing about all this is that Apache [2] not once reported the CGI (Common Gateway Interface) program crashing. To me, that seems fairly obvious, but apparently not. So there's no telling how long this has been happening (yeah, you can tell I check the logs often, can't you?).
I do need to think about how to handle command line arguments in crashreport(). It's library code and thus, shouldn't rely on global variables. I have to think about this …