I've spend the past few days battling a cracker on our system, a nice change (in an intellectual capacity mind you) over the typical script kiddies I've had to clean up after.
We were first made aware of the issue with the following ticket:
XXXXXXXXXXXXX XXXXXX XXX > Managed Security Service Provider > on behalf of XXX XXXXX
Subject: Notification of redirection site using a wildcard in your DNS (Domain Name Service) (Phishing)
Dear Sirs,
The Urls http://XXXXXXXXXXXXXXXXXXX, with the only exception of http://XXXXXXXXXXXX, which is a website hosted at IP (Internet Protocol) address XXX.XXX.XXX.XXX, all redirect on a copy (phishing site) of XXXXXXXXX site, one of the banks of our group. It appears that your name server (DNS) is using a wildcard to redirect any Url in the form:
http://[something].XXXXXXXXXXXXXXXXX
to a phishing web site located at:
http://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXX XXXXXX XXX, is assisting the group and its related entities in preventing or terminating any online activity that targets XXX XXXXXX clients for potential fraud. This activity violates XXX XXXXXX copyright, trademark and other intellectual property rights and may violate the criminal laws of the United States and other Nations.
E-mail messages have been broadly distributed to individuals by a person or entity pretending to be XXX XXXXX. These e-mails use XXX XXXXXX name and identity (including trademarks) without authorization.
The e-mails request recipients to verify and submit sensitive details related to their XXX XXXXX accounts. Within this message, there is a a hidden link that sends to a fraudulent web site displaying XXX XXXXXX copyrighted materials and trademarks.
The redirect mechanism shown above originates from a DNS server that is under your control. Its main purpose is to improperly obtain personal information of clients in order to illegally access their bank accounts.
The owners of these web sites typically perpetrate identity-theft related activities, such as using customers credit cards or bank accounts without authorization. Furthermore, since the vast majority of the e-mails are not sent to actual XXX XXXXX customers, these actions can damage the reputation and image of XXX XXXXX.
Please take all the necessary steps to immediately shut down the redirect web site, terminate its availability to the Internet and discontinue the transmission of any e-mails associated with this web site. We understand that you may not be aware of this improper use of your services and we appreciate your cooperation.
Thank you for your cooperation to prevent and terminate this fraudulent activity.
Sincerely, > XXXXXXXXXXXXX XXXXXX XXX > Managed Security Service Provider > on behalf of XXX XXXXX > Email XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
I won't say our DNS system is unique in the way it's set up. We have one master DNS server for all our zones (domains). The zones from this machine are pushed out to four DNS slave servers, two in one domain, the other two in another domain (and for illustrative purposes, I'll use example.net and example.org for the domains we use to resolve DNS for our customers). The four machines are split between two data centers, with one machine from each domain in each data center. The master is never queried from the internet; all Internet queries for our zone information go to the four slaves, and zones are pretty much equally divided between the two domains.
I check the master DNS server, and the zone in question has no DNS wildcard record [1]. I then check the slave DNS servers, and well:
>
```
$ORIGIN .
$TTL 598 ; 9 minutes 58 seconds
XXXXXXXX IN SOA ns1.example.net. root.ns1.example.net. (
2008101001 ; serial
10800 ; refresh (3 hours)
3600 ; retry (1 hour)
604800 ; expire (1 week)
598 ; minimum (9 minutes 58 seconds)
)
NS XXXXXXXXXXXXXXXXX.
NS XXXXXXXXXXXXXXXXX.
A 10.11.224.198
MX 30 XXXXXXXXXXXXX.
$ORIGIN XXXXXXXX.
mail A 10.11.224.198
old A 10.11.224.226
www A 10.11.224.198
* A 192.168.1.15
```
(IP addresses have been changed to private IP addresses for illustrative purposes.)
The 10.11.224.x addresses are ours, but that 192.168.1.15 isn't. I know there have been some recent attacks against DNS and I assumed that the cracker in question may have exploited DNS to add the record. I upgraded all our DNS servers to the lastest version of bind [2], fixed the zone and called it a day. The only services on the slave DNS servers is DNS and SSH, so those are the only two things that could be compromised.
The next day, a reply to the ticket:
We are sorry to inform you that the redirect we described in the ticket is still working.
We kindly ask you to take action as quickly as possible to terminate it.
Thank you,
...
The zone on the master wasn't modified; it was modified on the slaves. I poke around some more, and notice that the updates for the compromised zone are being rejected by the slaves. It turns out the cracker had protected that zone using the Linux command chattr [3] to protect the file from changes. And that can only happen at the command line.
XXXX!
I never liked the “Nuke and Pave” approach to security issues, since without knowledge about how they got in, how do I know I fixed the exploit? And given that the cracker had only modified this one file meant the attack was narrowly focused on propping up a phishing site. So I changed the extended attributes on the file to allow updates, made sure the zone data was fixed, and went about my business trying to figure out how they got in.
Next day, the wildcard DNS record was back, but changes on the master (and the master was never compromised, only the slave DNS servers serving up this domain) again weren't showing up on the slaves. The extended attributes on the file on the slaves where normal, but then I noticed the attributes on the directory! They were modified, and the chattr program (along with grep) were deleted from the machine.
Nice.
I never did figure out how they got in, and it appeared they were persistent enough to keep coming back (which was odd—I would think that they would realize the jig was up and move on; also odd was the zone they picked—one of the more popular websites we host (lots of pictures of bikini-clad models). So it was clear the only answer was to “Nuke and Pave,” but until I got a chance to do that, I needed to do something on a box the cracker has root access to.
So, while bind couldn't modify the file, I could edit it directly. After about fifteen minutes of thinking, I came up with what I think was the perfect response—I didn't delete the wildcard DNS record, I modified it:
>
```
* IN A 127.0.0.1
```
Rationale: Obviously, each time I deleted the record, they came back to “fix” it. So they must have some sort of automated process to check, and getting rid of it triggered that process. Here, I haven't removed the record, just changed it to something useless for their purposes [4]. I was working on the assumption of their automated monitoring process to just check for the existance of the record and not the actual contents, so they wouldn't come back.
They didn't.
And in the meantime, I did the “Nuke and Pave” patch to security on the slaves. And not only did I update bind, but sshd and severely restricted who could log onto the DNS servers.
[1] http://en.wikipedia.org/wiki/Wildcard_DNS_record
[2] http://www.isc.org/products/BIND
[3] http://en.wikipedia.org/wiki/Chattr