Ah, nothing like getting a call at the ungodly hour of 10:30 to fix a munged firewall. For reasons I'm still trying to figure out it apparently crashed last night and the fix was a power cycle.
But in looking over the log files, I kept seeing:
>
```
Jul 2 08:48:07 localhost kernel: SPOOF-SRC-172 IN=eth0 OUT=eth1
SRC=172.21.159.36 DST=XXX.XXX.XXX.XXX LEN=40 TOS=0x00
PREC=0x00 TTL=238 ID=0 DF PROTO=TCP SPT=11100 DPT=1516
WINDOW=0 RES=0x00 RST URGP=0
```
I had programmed the firewall to reject any private IP (Internet Protocol) address (since you're not supposed to see private IP addresses on the Internet at large) yet here I was, seeing a private, albeit a less known private, IP address. Only the source address kept changing for each packet logged, but always within the 172.21.0.0/24 block.
This meant that the address was originating somewhere on our network, but where?
Pinging one of the private IP addresses from the router directly connected to the firewall didn't show anything. It seems that the ping packets were sent to the main core router, which just dumps private IP addresses to the bit bucket.
But why was that router throwing anything from 172.21.0.0 to the firewall?
Now, the rule I have for filtering these packets in iptables looks like:
>
```
iptables --table filter --append FORWARD --source 172.16.0.0/12
--jump LOG --log-prefix "SPOOF-SRC-172 "
iptables --table filter --append FORWARD --source 172.16.0.0/12
--jump REJECT
iptables --table filter --append FORWARD --destination 172.16.0.0/12
--jump LOG --log-prefix "SPOOF-DST-172 "
iptables --table filter --append FORWARD --destination 172.16.0.0/12
--jump REJECT
```
which doesn't record the MAC (Media Access Control) address (which uniquely identifies an Ethernet interface on a network). However, by adding:
>
```
iptables --table mangle --append PREROUTING --source 172.16.0.0/12
--jump LOG --log-prefix "WHAT "
```
the MAC address is logged.
Cool!
>
```
Jul 3 17:48:24 localhost kernel: WHAT IN=eth0
OUT=MAC=00:10:e0:02:09:dd:00:02:85:12:a7:c0:08:00
SRC=172.21.117.49 DST=XXX.XXX.XXX.XXX LEN=40
TOS=0x00 PREC=0x00 TTL=239 ID=0 DF PROTO=TCP
SPT=11112 DPT=3716 WINDOW=0 RES=0x00 RST URGP=0
```
What's recorded is the MAC address of the interface (the first six octets), followed by the MAC address the packet came from (the next six octets) followed by the protocol (the final two octets, and in this case, it's an IP packet).
Now, the first three octets of a MAC address are the vendor code—who makes that particular Ethernet device. Easy enough to look up [1] and—
A Riverstone?
It's the router?
There's nothing on our network that uses the 172.16.0.0/12 private IP space. And yet, here's one of our routers, sprewing forth garbage packets.
Sigh.
Time to get with Dan the Network Engineer on this …
Between Dan the Network Engineer and I, the problem was solved [2].