“I've got some bad news,” I said to Smirk.
“You know the rules,” he said. “You have to make it sound like good news.”
I thought for a few moments. “We can save electricity by turning off this server,” I said, pointing to one of the boxes in the office.
“It doesn't work?”
“No,” I said.
“It was hacked?” asked Smirk.
“'Fraid so,” I said.
* * * * *
I only noticed because the network to the office went down. I could ping other boxes in the office, but couldn't get past the firewall. I then went into the data center, logged onto one of the servers and yes, I could still get out to the Internet (relief there). But I couldn't ping the firewall from outside.
I then checked the firewall, and it was still up and running. If I unplug the office network, I could ping the firewall from the outside. Plug the office network in, and it gets swamped.
So something on our network was spamming the network.
Spent the next few minutes plugging and unplugging various bits of the office network until I isolated the culprit—the system that monitors the servers and network.
Of course!
It was at that point that I broke the news to Smirk.
* * * * *
Once it was unplugged, I start poking the machine and yup, there was this odd process: “./s 202.XXX.XXX.XXX”. It was pretty easy to locate the actual executable under /dev/shm/http/, which is normally not a place for executables. The process was running as the apache user, and the files under /dev/shm/http were owned by apache, which to me, is a rather obvious clue that it was Apache [1] was the vector of the exploit.
Among the files in /dev/shm/http/ were configuration files to a hacked IRC (Internet Relay Chat) bot [2] (which was named s) and some other program (named httpd—not sure what that was) and a few scripts to start things up and to clean the log files (which the cracker had not bothered to run). The hacked up IRC bot would just sit there until commanded to participate in a DDoS (Distributed Denial of Service) attack, which obviously was the cause of the sudden network activity, aimed at a site in Japan.
But how? How did this person get in?
More poking around the system lead to a rather curious request just a few minutes before I lost network connectivity (due to the firewall being swamped)—it was a request to Cacti [3], a data storage and graphic package with a web based interface written in PHP [4] (why does that not surprise me?). It seems we had fallen prey to a Raxnet Cacti graph_image.php Remote Command Execution Exploit [5], which basically means the cracker was able to send a command to the server, in this case, a command that would download a perl script and execute it, opening up a shell to a remote connection. It was through this that the hacked programs were uploaded and started.
Fortunately, this is the only system running Cacti, and second, there was no trust mechanism to any other machine on the network from this machine. And third, this is a machine that we only log in to, never log in from, so any damage was limited to just this machine.
On the bright side, it could have been worse [6].
[2] http://www.energymech.net/
[5] http://www.frsirt.com/exploits/20050911.cacti_graphimage_exec.pm.php