Well, I did not go to Miami [1] today because of server problems last night (or technically, early this morning). The purpose of the trip to Miami was to retrieve two (of the four) servers I admin in order to install Gentoo [2] and get rid of this silliness called RedHat [3].
The other admin ended up going down to Miami anyway and delivered the servers to my door step. Later on in the evening, Mark [4] came over to help me with my first Gentoo installation. Gentoo is pretty neat. A “stage 1” [5] installation (which we did) took several hours to perform, as it installs a base configuration, then downloads and recompiles everything (given the compiler options specific for the particular architecture for best performance). You can also specify what you want and more importantly, don't want.
Another interesting feature is that the base system allows you to log in via ssh so Mark and I spent most of the time outside in the courtyard watching the installation via the wireless network here, and discussing various issues.
One of which was the constantly crashing server. Mark mentioned that he had encountered a similar problem on a friend's webserver, due to the web log files never being rotated. Well, the server from hell [6] has that problem in spades—over 1,000 sites and none of the logs have ever been trimmed. And we're talking both the access_log and the error_log files.
Now, I had discussed the error_log situation with the client—namely does each site really require its own error_log? The client agreed with me that no, each site did not need said file. So after Mark left, I proceeded to nuke all the error_log files (since really, it's only used to debug CGI scripts and even then, that's not a common thing). That alone cleared up some 12 gigs of disk space. I then rotated all the access_log files so now hopefully that server won't crash.
[5] http://www.gentoo.org/doc/en/handbook/handbook.xml?part=1&chap=2#doc_chap2