Mark's [1] webserver Seminole [2] has been having problems running on tower. The architecture of Seminole is multi-threaded, but the threading library under RedHat [3] 5.2 (and Linux 2.0) is basically broken, so Mark is having to simulate that by using fork(). And it's not working that well. It'll run for a few days, then the process accepting new connections will die for no apparent reason, leaving the main process hung since it's not the one accepting connections and it isn't aware that its child process died.
Mark, being primarily an embedded systems programmer [4] isn't that well versed in all the picky details of POSIX (and especially signals—granted it is a mess) so I wrote some code to trap every possible signal [5], log it to syslogd and then exit. Hopefully we'll be able to track down what might be happening.
I also pull out the state of the registers but that is very dependant upon the compiler options used and is a complete hack (that as far as I know, only works under the Intel platform without the -fomit-frame-pointer option under the Linux 2.0 kernel.
Did I mention it being a hack?
[1] http://www.conman.org/people/myg/
[2] http://gladesoft.com/products/seminole/