The problem this morning [1] was a direct cause of my inability to fully grok iptables. I logged into the customer's firewall (we offer managed firewalls as one of our services), which was also running an instance of Cacti [2] to help monitor their network. Sure enough, the SNMP (Simple Network Management Protocol) polling script was failing for some obscure PHP reason.
Poking around the system, I found a few suspicious files, time stamped two weeks ago, named ping, ping.1 and ping.txt. Odd, I thought and when I checked the contents, yup—a script kiddie script, which opens up a connection to a remote computer.
Sigh.
More poking around, and I find rather quickly the IRC (Internet Relay Chat) bot program the script kiddie was running (all files owned by the webserver).
Okay. Cacti has some … issues … with security, and it's no surprise that the script kiddie … exploited … these issues, to install their nefarious wares. And the network latency the customer was experiencing was due to excessive IRC traffic.
The major problem I had was how the script kiddie got access to the webserver in the first place. Due to Cacti's … issues … with security, I had explicitly blocked access to all network services with iptables (with the exception of traffic from The Office). Only, what I thought I did, and what I actually did were two different things (much like in practice how theory and practice differ). I spent several fruitless hours (including blocking all traffic to the firewall itself but not through the firewall, which made the remote administration … difficult) before buckling down and really reading up on how packets flow through iptables.
Now, I had set this up to match our office setup. The only real difference (and it's a major difference) is our Office Firewall doesn't NAT (Network Address Translation), but our customer's firewall does. Oh, that, and we don't run any services on our firewall. Two, two major differences between our Office and the customer are our lack of NATing, services, and an understanding of iptables. Our three major differences between … oh, I'm digressing.
About an hour and several hand drawn diagrams later, I finally had a grasp on the flow of packets through iptables:
[Flow of packets through IPTables] [3]
I had the filtering rules in the wrong place, along the packet forwarding path (right hand side of the diagram) instead of the local interface input path (bottom half of the diagram). Once I solved that little problem, then I could concentrate on removing the IRCbots and fixing Cacti (I'm guessing the exploit causes Cacti to stop functioning properly—easiest fix was to reinstall Cacti and make sure I had the file permissions correct).
![[3] /boston/2007/01/31/iptable.gif](/gemini.conman.org/boston/2007/01/31/iptable.gif){kind=link}