Watching monnet again I see some odd activity coming from an IP address. Random TCP packets with the Reset bit set to random TCP ports on my primary machine. I try to trace back the connection and it goes nowhere, so the source address seems to be forged.
I might have to talk to my upstream provider on what to do.