“In a system of a million parts, if each part malfunctions only one time out of a million, a breakdown is certain.”
“Stanislaw Lem”
“Project: Leaflet” is a web based application written in PHP, so there's a dependency on a webserver (in our case, Apache [1]) and PHP [2] itself. There's also a database involved, so throw in MySQL [3], and, oh heck, Linux to run the whole mess, and you have a pretty standard LAMP (Linux, Apache, MySQL, PHP/Perl/Python) stack going on.
Although in our case, we use PostgreSQL [4] (and yes, I have two active versions [5] of “Project: Leaflet”) so that means we have a pretty standard LAPP (Linux, Apache, PostgreSQL, PHP/Perl/Python) stack going on.
Now, “Project: Leaflet” is designed to help manage mass emails, like a newsletter, or a product announcement, or perhaps to notify customers of an impending company name change [6], so it requires an SMTP (Simple Mail Transport Protocol) server of some sort. At first we used whatever SMTP server came with the distribution of Linux we used (at first sendmail [7] but now we also use postfix [8]).
Then came the automatic handling of bounced emails, which meant a bit more involvement with the SMTP server (sending emails to a particular address to a program). It also meant parsing the incoming message and given that there really isn't a standard for bounce messages (or rather, quite a few, take your pick) the stock configuration for the SMTP server wasn't going to cut it anymore.
First off, we needed to enable address extensions [9], and at least under postfix, that's just adding one line to the configuration, but it's still a change from the default install (such a configuration may be enabled by default under sendmail, but we don't run “Project: Leaflet” with sendmail, so it's a moot point). The point of the address extension is to encode the outgoing email address in our return address so we don't actually have to parse the body of the bounce.
It also helps with handling confirmation replies.
But there's still the issue of parsing the headers from the incoming email, and well, I know procmail [10], and I know that's what it's there for, to help in dealing with incoming emails. I mean, it's there right? (Well, actually, not by default any more. And that means yet more changes to the SMTP configuration to enable procmail processing). And yes, the syntax is horrible, but it's the very model of clarity compared to some other syntaxes you come across on a Unix system (like, oh, sendmail.cf [11] anyone?).
So now the dependencies for “Project: Leaflet” are: Linux, Apache, PostgreSQL, PHP, sendmail or postfix (with some custom configurations) and procmail.
Oh, and then there's the change from a few weeks ago, where I made it such that a single copy of the code can serve multiple instances (clients) on a single server. I did that not to save disk space (heaven's no! The uncompressed source code doesn't even come close to a megabyte of disk space) but administrative overhead—I can update the code in one location per server instead of, say, 25 individual copies.
But that necessitated some changes in layout and assumed locations of files and well … let's just say that while a company could take our code base (and at one point the topic of releasing the code as Open Source™ did come up), it's less and less likely that an average website owner would be able to install this program, given the dependencies and somewhat custom configurations made thus far.
Now, this version (the “Install Once And Use Everywhere On A Server” version) mandated yet another “feature”—support of a per-client IP (Internet Protocol) address.
We have the IP addresses. No problem there.
And Linux supports multiple IP addresses. No problem there.
And Apache supports multiple IP address. No problem there.
sendmail/postfix support multiple IP addresses. No probl—
Uh … hold on a second …
Er, mu [12].
Incoming email isn't an issue—sendmail/postfix can be set up to listen on all interfaces. It's outgoing email that's an issue.
See, it's all too easy for certain ISP (Internet Service Provider)s to blacklist IP addresses that send an excessive amount of spam (and that happens to us all the time [13]), so if that happens, we want the customer's assigned IP address to get banned, and not the IP address of the server (otherwise, all our customers on said server have “an issue”).
But, when sendmail/postfix send email, they use the default server IP address, which isn't what we want.
So, mu.
After some searching, we found that exim [14] will do what we want (great! Yet another SMTP server we need to learn and support), but not out of the box (nor will it support address extensions, virtual hosts or procmail, so there're quite a bit of configuration changes going on).
So, this little PHP app I wrote requires Linux, Apache, PHP (of course), PostgreSQL, exim and procmail. Talk about your debugging nightmare.
Today, I spent several hours tracking down an issue where emails were being sent out via the server's primary IP address and not the client's IP address. That meant I had to track the issue through half the chain there, PHP, exim and procmail. I ended up chasing (now that I look at it after the fact) a potential red herring [15]. The problem was compounded by the fact that it worked for the test account.
I fixed the issue by replacing the call to the PHP mail() function (which calls the external program sendmail—which isn't really sendmail but a simple replacement provided by exim, which injects the email directly into the outgoing queue) with a bunch of PHP code that talks to the local email server via SMTP.
That worked. Only later, I realized that the client was using the incorrect email address for sending emails, which was probably why the primary IP address was showing up when it shouldn't (and if that makes no sense to you, Welcome to My World!).
It certainly feels as if the modern web development world is held together with chewing gum and bailing wire (and we're all out of chewing gum) and nobody really understands what's going on underneath the covers (and frankly, are as scared of it as they are of seeing how sausage is made).
[4] http://www.postgresql.org/
[9] http://en.wikipedia.org/wiki/E-mail_address#Sub-addressing
[11] /boston/2009/03/24/sendmail.cf
[12] http://catb.org/jargon/html/M/mu.html