So much for a quiet day at the office.
A little background: a while ago I set things up so that all the email that the various programs send out, on all the servers we have, gets funneled into a single account, which is then forwarded to me for review: output from cron, the mailscanner software, bounces, etc.
Yes, it generates a large amount of email, but mostly it's stuff I can quickly scan and dump as all I'm really doing is looking for anything out of the ordinary. Except for the bounce or non-deliverable messages, bitching about the inability to send emails to non-existant accounts across the internet (XXXXXXX spammers) and for those, I funnel those into a single folder (using procmail) which I then delete on a daily basis (why don't I just delete them as I get them? There is the rare occasion when I need to check those messages and because of that, it's easier to just delete them manually than recofiguring the email client when I need to actually search for those messages).
So it was surprising to find such messages in my default inbox. More surprising was the sheer number: over a thousand.
And not only in my default inbox, but in just about every incoming email folder. Hundreds. Thousands. Jamming up my email.
Checking the procmail log file, I found thousands of instances of:
>
```
procmail: Error while writing to "in-mailerdaemon"
procmail: Truncated file to former size
procmail: Error while writing to "in-mailerdaemon"
procmail: Truncated file to former size
```
Hmmm …
The folder itself was 51,200,000 bytes in size. Inside were nearly 6,000 messages, all the bitching about the same thing:
>
```
----- The following addresses had permanent fatal errors -----
<root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
(reason: 554 5.4.6 Too many hops)
<admin@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
(reason: 554 5.4.6 Too many hops)
----- Transcript of session follows -----
554 5.4.6 Too many hops 26 (25 max): from <root@XXXXXXXXXXXXXXXXXX>
via localhost, to <root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
```
which was trying to inform me that it couldn't deliver the following email message:
From: root <root@XXXXXXXXXXXXXXXXXX>
To: admin@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Subject: CRITICAL:Current disk space consumption has reached a critical limit.
Date: Sat, 18 Nov 2006 04:23:27 -0500
Cc: root@XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
X-Yoursite-Mailscanner: Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details, Not scanned: please contact your Internet E-Mail Service Provider for details
X-Yoursite-Mailscanner-Information: Please contact the ISP for more information
X-Mailscanner-From: root@XXXXXXXXXXXXXXXXXX
What's even funnier is that this is a customer of one of our resellers. Even funnier is that the site in question has been suspended for non-payment.
Wheee!
So I delete the overflowing inbox, and the incoming email starts flowing properly again, and I start investigating the problem further.
A few hours go by, and I start finding bounce messages in the wrong inboxes. Same error. And the folder is again 51,200,000 bytes in size.
Gee, guess I found some internal procmail limit or bug or something.
In the meanwhile, I find out that not only does the site belong to one of our resellers, and has been suspended, it has also moved servers!
Wonderful!
And the mail is originating from the original server, bouncing around the new server and after bouncing around too many times, bounces towards me.
Now, have I mentioned I hate control panels?
Yup, they show up here, because the makers of Insipid have modified sendmail [1] to handle their method of virtual servers and the command I'm used to using to check mail queues, mailq, is broken. So I have to track down where on God's Green Server they stuck the outgoing email queue and clean it up on two servers.
And yet I'm still getting bounce messages from the server, dating back to Saturday!
It was then I realized that our Office Email goes through yet another server, a spam firewall appliance. And it probably has a huge backlog of messages headed my way. And unfortunately, there's no way to flush the outgoing queue, because we have no access, because it's an appliance.
Sigh.
So for now, I have the email directed towards the bit bucket. I just hope that for the next few days, I don't miss a really important message.