While looking for referer spam [1] I found that my entry about fending off a DDoS (Distributed Denial of Service) attack [2] has been quite popular, but upon rereading I found it rather terse. So I figure I might as well clear up some details of what exactly I did.
netstat -an to check the state of all network connections, and given that there were an enormous number of connections in the SYN_RECV state is an indication that a SYN flood (where hundreds of connections are initiated but not completed, thus flooding out legitimate traffic) is underway.
Normal TCP/IP (Transmission Control Protocol/Internet Protocol) networking is open to an attack known as “SYN flooding”. This denial-of-service attack prevents legitimate remote users from being able to connect to your computer during an ongoing attack and requires very little work from the attacker, who can operate from anywhere on the Internet.
SYN cookies provide protection against this type of attack. If you say Y here, the TCP/IP stack will use a cryptographic challenge protocol known as “SYN cookies” to enable legitimate users to continue to connect, even when your machine is under attack. There is no need for the legitimate users to change their TCP/IP software; SYN cookies work transparently to them. For technical information about SYN cookies, check out <http://cr.yp.to/syncookies.html [3]>.
If you are SYN flooded, the source address reported by the kernel is likely to have been forged by the attacker; it is only reported as an aid in tracing the packets to their actual source and should not be taken as absolute truth.
Help from the Linux Kernel 2.4 Configuration screen
sysctl -w net.ipv4.tcp_syncookies=1 is the command used to enable SYN cookies in the Linux kernel. This helps some with the type of attack we were experiencing.
sysctl -w net.ipv4.tcp_max_syn_backlog=2048 increases the number of incomming connections the kernel can keep track of. Increasing this value is a bit of a double edged sword in such an attack—on the one hand, we allow more connections, thus hopefully allowing legitimate connections through, but on the other hand, we allow more connections, thus allowing more machines to SYN flood the machine. Given some of the other steps I took, this was probably a good idea overall.
sysctl -w net.ipv4.tcp_syn_retries=2 (which I forgot to mention in the original entry) decreases the amount of time the kernel spends trying to establish a TCP/IP connection (from a default value of 5 attempts to two) which helps to flush the bad connections from the system quicker.
route add -host <ip-addr> reject which causes the kernel to ignore packets from the given IP (Internet Protocol) address, and also flushes current connections from said IP addresses from the system. This was the thing I was doing that kept the system up and running during the attack. I ended up writing a script to continuously check the connections, then once a certain threshhold of bad connections was exceeded, ban all the addresses.
The site was eventually taken down dispite all the attempt I made to keep it up since the network traffic to the site in question was swamping the rest of the network the machine was on (it was the colocation facility that said enough is enough and shut the site down). Other than that, I was fairly successful in keeping the website accessible.