Ah, the joys of getting up the ringing of a cell phone. “Sean,” said R, who owns the servers I'm admining, “the site is down.”
“Mwuggua,” I said.
“Please, check it out,” said R.
“Umyeaokay,” I said, rolling out of bed. I make my way to the Computer Room, ping the backup server. It's alive. I log in. I log in. I log in. It finally sinks in that I was able to log in. And the system load is low too. I then try to bring up a webpage.
Nothing.
Doing it by hand, I see that the web server appears to be wedged. I do a netstat -an and see hundreds of connections in the SYN_RECV state. Okay, I think as I consume the Elixer of the Gods—Coca-cola. Lots of sockets bound up. Need to reset the webserver. The second I restart it, hundreds of SYN_RECV connections. Looks like a SYN flood.
With some help from Mark [1], I tweak some network variables: sysctl -w net.ipv4.tcp_syncookies=1 and sysctl -w net.ipv4.tcp_max_syn_backlog=2048 and restarting the web server helped a bit. Mark then had the idea of rejecting the attacking IP (Internet Protocol) addresses with route add -host <ip-addr> reject which helped even more (with a script to automatically do that). Then it was a matter of checking to see if there were too many attacking IPs, then running the blocking script. Yet another script to automate that and the site can still be accessed while under attack.
But that still means the site is under attack and all that traffic from hundreds of machines (at least 500, possibly more) is still flowing across the network, causing havoc. And I doubt it's going to get easier [7] any time soon (the company who's sites are being hosted were already extorted last year—this seems to be a different group … they think).
There isn't much that can be done about a DDoS (Distributed Denial of Service attack) since most of the attacks now a days are done via compromised machines across the Internet (I recorded attacks from machines from Asia, Europe, the Middle East, South and North America) that basically, you have to prepare for a slashdotting [8] if you want to survive a DDoS, and hope that your provider doesn't kick you out for repeated attacks.
Why I did what I did during a DDoS attack [9]
[2] http://www.joewager.com/extortion.htm
[3] http://www.joewager.com/extortion-hollywood.htm
[4] http://www.joewager.com/hackers_and_extortion.htm