gemini://gemini.conman.org/boston/2006/01/11.1
As if yesterday weren't bad enough [1], today I come home to find The Younger offering much sacrafice to the Great Porcelain God, due to stuff he may have ingested. Wlofie had called Poison Control and while what he ingested normally wouldn't be a problem, since he was making much offerings to the Porcelain God and complaining about abdominal pains, it was suggested we head off to The Emergency Room!
[It was an emergency! I didn't have my camera on me, so I had to make due with what was available—which was Wlofie's phone] [2] [3]
We arrived just before 8:00 pm, with Spring [4] flying to The Emergency Room from work. She arrived around 8:30, and shortly thereafter (after yet another offering to the Great Porcelain God) The Younger, accompanied by Spring, went into the bowels of the hospital.
At 1:30 am (yes, I am pre-dating this, as most of this happened Wednesday, and had there been wireless at The Emergency Room, I would have posted, but alas, there was none) Spring called, saying that The Younger's X-ray was inconclusive and next up was a CAT (Computerized Axial Tomography) scan, and that the rest of us should probably head on back home as there was no telling how long The Younger and Spring would be there.
And that, alas, is all the information I have at this time.
So … how was your day?
Doctors think it's a stomach flu, although why it took a CAT scan to figure that out, I don't know.
[2] /boston/2006/01/11/thumb.emergency.jpg
[3] /boston/2006/01/11/emergency.jpg
gemini://gemini.conman.org/boston/2006/01/12.1
Today was spent at The Hospital.
Not The Hospital whose Emergency Room [1] I dropped The Younger off at. No. That Hospital does not have a pediatric section, so The Younger was transported to a different Hospital during the wee morning hours.
Now, since Spring [2] has already mentioned one reason why he's there [3], I no longer have to dance about the issue of The Younger swallowing two kitchen magnets.
Magnets.
Yes, plural.
Now, as kitchen magnets go, these are tiny. Think of a push pin, one used to tack notices up to a bulletin board, but instead of a metal pin sticking out of it, it has a small round magnet to hold it to a refridgerator. We're talking something about 1/8″–1/4″ across.
And he only swallowed the magnet part, not the push pin part. Just the magnet part.
It was suggested by the doctors last night that The Younger might have pica [4] which is a persistent craving for non-food like items.
Oh, and he also got a stomach flu, as if things weren't bad enough.
So anyway, today was spent at The Hospital.
The Younger was sent through X-ray enough times that he no longer needs a nightlight and the reason for the multiple X-rays throughout the day was to track the movement of the two magnets [5]. The concern was if the two magnets attract each other in different parts of the intestines and lodge themselves against the intestine wall. Fortunately, the X-rays showed that not only where they progressing down his system, but they had already attracted to each other and travelling as one unit if I understood it correctly.
So, once the magnets are flushed from his system, The Younger can then return back home.
[3] http://www.livejournal.com/users/springdew/394438.html
[4] http://kidshealth.org/parent/emotions/behavior/pica.html
[5] http://en.wikipedia.org/wiki/Lemmiwinks
gemini://gemini.conman.org/boston/2006/01/12.2
Since The Younger and Spring [1] were transported to The Hospital from The Emergency Room [2] via ambulance, Spring's car was still at The Emergency Room. I worked out a plan whereby Smirk would pick me up at The Hospital, drive me to The Emergency Room, whereby I would drive Spring's car back to the Hospital, then I would drive back home to get some sleep so I could go to work tomorrow.
Around 11:30 pm or so, Smirk calls me on my cell phone to say he's outside the main entrance to The Hospital. The main entrance to The Hospital is this large revolving door (it has motion detectors, and will start moving automatically). I nearly smack into the thing trying to walk outside.
It's not moving.
I try pushing it on the sign that says “Don't Push” but it won't budge. In fine print, the sign mentions something about the handicapped doors to either side of the revolving door. So I pick the one to the right, and try to go out.
It doesn't budge.
Nearby is a large flat disk with a stylized wheelchair and the words “Open Door” on it (or something to that effect—I don't recall exactly). It's made to be pressed, and I press it. I hear this light buzzing noise, only the door isn't opening. I try pushing the handicapped door, but it still doesn't budge.
Neither does the one to the left of the revolving door.
Through the windows, I can see Smirk's vehicle.
I try all three egresses again, pushing and shoving and otherwise trying to get to the other side of the doors.
Then I call Smirk on the cell phone, to inform him that I'm having some difficulty in leaving The Hospital. Through his laughter I'm told that he's seen my difficulty and suggests meeting him outside this Hospital's Emergency Room.
gemini://gemini.conman.org/boston/2006/01/13.1
As if Tuesday [1] wasn't bad enough (so was Wednesday [2] and Thursday [3] but that dealt with things outside of work), I arrive at The Office to one machine spewing out a sustained 20Mbps (Megabits per second) of traffic, and another one not responding at all.
For the one vomitting over the network (a customer's colocated box), I logged in, and saw the following errors:
>
```
eth0: Transmit error, Tx status register 82.
Probably a duplex mismatch. See Documentation/networking/vortex.txt
Flags; bus-master 1, dirty -1756485798(10) current -1756485798(10)
Transmit list 00000000 vs. f7ed8480.
0: @f7ed8200 length 80000042 status 00010042
1: @f7ed8240 length 8000002e status 0001002e
2: @f7ed8280 length 800005d6 status 000105d6
3: @f7ed82c0 length 80000548 status 00010548
4: @f7ed8300 length 80000036 status 00010036
5: @f7ed8340 length 8000003e status 0001003e
6: @f7ed8380 length 80000042 status 00010042
7: @f7ed83c0 length 800005ea status 000105ea
8: @f7ed8400 length 800005ea status 000105ea
9: @f7ed8440 length 8000004e status 8001004e
10: @f7ed8480 length 800005d6 status 000105d6
11: @f7ed84c0 length 800005d6 status 000105d6
12: @f7ed8500 length 800005d6 status 000105d6
13: @f7ed8540 length 800005d6 status 000105d6
14: @f7ed8580 length 800005d6 status 000105d6
15: @f7ed85c0 length 80000042 status 00010042
```
Over and over again, for days—ever since it was rebooted last week. So I check Documentation/networking/vortex.txt:
## Transmit error, Tx status register 82
This is a common error which is almost always caused by another host on the same network being in full-duplex mode, while this host is in half-duplex mode. You need to find that other host and make it run in half-duplex mode or fix this host to run in full-duplex mode.
As a last resort, you can force the 3c59x driver into full-duplex mode with
> > options 3c59x full_duplex=1
>
but this has to be viewed as a workaround for broken network gear and should only really be used for equipment which cannot autonegotiate.
I know Dan the network engineer prefers not to have auto negotiation as he's seen problems with it, so I call him up, and ask him to set the switch port the machine is plugged into as 100Mbps auto-negotiate. Once he did that:
>
```
eth0: Setting full-duplex based on MII #24 link partner capability of 4101.
```
And the traffic immediately cleared up (I suspect that when the box was first plugged in, the switch port was set to auto-negotiate, but some time afterwards, Dan set the switch port not to negotiate, but since the negotiation already happened, no errors were reported. It was only after the box was rebooted last week that the problem cropped up).
The second box was also a colocated machine, and the customer there just told me to power cycle it. Instead of just powercycling the machine, I hooked the crash cart up to it, just to see if any error messages were on the screen.
Only the screen was blank.
Couldn't get it to unblank at all.
The machine must be hosed, so I power cycled it.
And that's when I saw the other video interface.
Sigh.
The rest of the day was pretty much like that—just as I clear one problem, another would pop up. All day I was busy with problems.
It was like [DELETED-Monday-DELETED] Tuesday [4] all over again …
gemini://gemini.conman.org/boston/2006/01/13.2
Labrea [1] is actually logging about half a gig a day [2]. Over a 24 hour period (from about 6 am Thursday to 6 am today) I'm tarpitting 82,359 connections across 2,059 unique IP addresses (24,252 connections from a single IP address). And while the number of network ports being accessed has increased a bit, it's the Microsoft specific ports that are still the most popular targets (with 72% of the scans):
Table: Top 10 ports captured by Labrea in the past 24 hours Port # Port description # connections ------------------------------ 139 NetBIOS (Basic Input/Output System) Session Service 24941 445 Microsoft-DS (Directory Service?) Service 23013 1433 Microsoft SQL (Standard Query Language) Server 6772 4899 Remote Administration [3] 5620 135 Microsoft-RPC (Remote Procedure Call) service 4722 80 Hypertext Transfer Protocol 3697 8080 Hypertext Transfer Protocol—typical alternative port 1686 7212 (unknown) 1683 8000 (unknown) 1471 10000 (some web based control panels use this port) 951 ------------------------------ Port # Port description # connections
The program I'm using to generate the stats is written in Perl, and it took about 4 hours to run over a day's worth of data (the machine that does the tarpitting isn't the fastest machine we have, but it's more than enough to dedicate to just running LaBrea). I definitely want to write a program to process LaBrea data in real time.
[1] http://sourceforge.net/projects/labrea
gemini://gemini.conman.org/boston/2006/01/14.1
I'm at the hospital [1], which unfortunately does not have Internet access (which sucks, and it's obvious that I'm posting this some days after I wrote this).
Good news for The Younger: he seems to have gotten over his stomach flu.
Bad news for The Younger: the magnets have been stuck somewhere in the large intestines for the past 48 hours. And he won't be released until the magnets are released.
Sigh.
gemini://gemini.conman.org/boston/2006/01/14.2
Since I'm stuck at The Hospital [1], and The Hospital doesn't have wireless, I might as well work on the real-time LaBrea data processing program. Loaded up the laptop with program fragments and libraries I may need and brought along my copy of Advanced Programming in the Unix® Environment [2] by W. Richard Stevens, and am coding up a storm.
And yes, I'm coding this up in C. Not like I'm going anywhere in the next couple of days.
* * * * *
LaBrea spits out lines like:
>
```
1136832897 Initial Connect - tarpitting: 82.240.204.251 3334 -> XXX.XXX.XXX.XXX *
1136832897 Initial Connect - tarpitting: 82.240.204.251 3339 -> XXX.XXX.XXX.XXX 139
1136832897 Persist Trapping: 82.240.204.251 3334 -> XXX.XXX.XXX.XXX 445 *
1136832898 Persist Activity: 216.248.36.242 45285 -> XXX.XXX.XXX.XXX 135
1136832898 Persist Activity: 216.248.36.242 34589 -> XXX.XXX.XXX.XXX 135 *
1136832898 Persist Activity: 216.211.61.158 3862 -> XXX.XXX.XXX.XXX 135
1136832898 Persist Activity: 211.236.205.138 4459 -> XXX.XXX.XXX.XXX 139 *
```
There's some other stuff that's logged, but I'm primarily looking for lines like the above. Which means the information I can store will look something like:
>
```
struct tprecord
{
IP src;
Port sport;
IP dest;
Port dport;
time_t start; /* first time we see src:sport-dest:dport */
time_t last; /* time of last activity of src:sport-dest:dport */
size_t packets;
};
```
Now, since we'll be handling a lot of connections, we need to store these structures in a way that is easy to search through. Could use a hash table, and we are keying off the src:sport→dest:dport tuple, and that's 10 bytes of pretty much random binary data, which is good for this type of thing. But it's an open ended hash, not knowing how many entries we'll be handling. There are several methods to handle overflows in a hash table, but that still means at some point doing a linear search, and then there's the overhead of having a hash table.
There's also storing the data in a tree. In fact, I even have code in C to manage a balanced tree (taken from Knuth). But the downside is—I only have code to add to the tree, not to delete from it. And it was hard enough to write as is.
And there's still the overhead of maintaining a tree structure.
So I'm using arrays. Less overhead and straightforward to manage. The only trick is keeping the array sorted in some order, but as long as you compare the structures in a consistent manner, that's not really an issue (and by the way, I'm sorting by source address, port, destination address and port, in that order).
To make it even easier, I have an array of structures (see above), and an array of pointers to said structures, and it's the pointer based array that is sorted (since it's faster to swap pointers than swap whole structures). And once sorted, you can use a binary search to find the record.
The default binary search in C, bsearch(), returns either a pointer to the found record, or NULL if not found. Good for most uses, but not quite what I want. In addition to either finding the record or not, I'd also like to know where in the array the record was found, and if not, where it would appear if it was in the array. And C's bsearch() does not return the index.
Why do I want that?
My thinking is that if the search (reguardless of what type of search it is) didn't find the record in a sorted array, it has the information as to where the missing record should be. And with the information as to where, it would be “trivial” to add the record at the right point in the array.
And that beats adding the new record at the end, and restorting (C's qsort() is Quick Sort, which on average is O(n lg n) running time, but the worse case is O(n^2) and the worst case is then the array is mostly sorted to begin with, which is why I'd rather not resort after each addition).
And try as I might, I could not get a binary search to work that would also return the index. For now, I replaced the search with a linear search, until I can comb through some reference materials at home.
The other problem I'm having is the array manipulation code doesn't quite work. When the program starts, it has space to store X records. If it runs out of space, it increases the size of the various arrays until it hits some large number of records, then it will go through, purge records fitting some criteria.
Only it's not working properly.
Got a lot of work done on the program though, and it was nice to get into The Zone (even with a headache). At this point, I'm headed home for sleep and what looks to be yet another day at The Hospital.
[2] http://www.amazon.com/exec/obidos/ASIN/0201563177/conmanlaborat-20
gemini://gemini.conman.org/boston/2006/01/15.1
Back at The Hospital [1]. The Younger is over definitely over his stomach flu, but the magnets are still stuck.
Normally on Sundays I run a D&D (Dungeons and Dragons) game for the family and The Kids were looking forward to playing this week, but frankly, I don't want to run a session at The Hospital. For one thing, I have a pile of reference material I use during a session and there isn't a good enough surface for me to use. And the smoke breaks (for both Spring [2] and Wlofie smoke) will be longer (since they have to head outside The Hospital instead of just outside Casa New Jersey), leading to less actual gaming time.
Second, I'm still in a programming head space [3] and don't want to loose being in The Zone.
Third, Spring fell asleep as soon as I got there, and Wlofie was home sleeping, and that's half the party. I'd rather not run a session with that many missing party members.
* * * * *
In an effort to get the magnets moving, The Younger is being forced to eat (not only by us, but through The Hospital staff) even though he complains of stomach pains.
I have this feeling that it's going to be a long hospital stay.
gemini://gemini.conman.org/boston/2006/01/15.2
You would not believe how hard it was to write a binary search that returned the correct index for a missing record in an array.
Even with Programming Pearls [1] by Jon Bentley (reference book of the day).
Binary search is a deceptively simple algorithm that is easy to get wrong (be especially careful when searching empty arrays). And of the binary search implementations I've seen, when they do return the index in the array, it's to an element that's actually there, and some other value (like 0 or -1) if the element isn't there. I've yet to see an implementation that returns an index reguardless (plus an indication if the element was found or not).
So I spent several hours getting a binary search to return an index even if the element wasn't found. Before looking at my solution [2] you should at least try coding it (I should note that the code is pulled from the real-time LaBrea data processing program I'm writing, which just enough changes to get a single file to compile and with as little change to the source code as possible, so some of the names may not make that much sense, but the logic should be clear).
[1] http://www.amazon.com/exec/obidos/ASIN/0201657880/conmanlaborat-20
gemini://gemini.conman.org/boston/2006/01/15.3
I've been simplifying the real-time LaBrea data processing program [1]. One simplification: don't try to grow the arrays. If I'm trying to grow the arrays and I'm going to stop at some point anyway, why not just allocate as much as I'll ever use when the program starts? It gives you a definite upper bound (and one that can be set from the command line) and boy, does it simplify the code.
I also got the binary search routine [2] working correctly after a few hours.
And I had to remind myself just how C's qsort() works, and make sure I know that (in my program at least) I'm getting pointers to pointers to structures, and not just pointers to structures (when I fixed that, I started getting correctly sorted arrays).
The only outstanding bug I have right now deals with adding a record after the array has been purged of old records. The problem is that I'm using an index from before the purge. I should be able to get this running on the LaBrea tarpit system in the next day or two.
gemini://gemini.conman.org/boston/2006/01/16.1
The Younger has yet to pass the magnets, so he's still in The Hospital.
Sigh.
And so far, despite repeated corrections, most of The Hospital staff still refer to The Kids as “girls.” Okay, understandable given the shoulder length hair both have, and slender bodies.
But repeatedly?
“So, is she with you?”
“Yes, he is.”
“Should I give her a badge?”
“Yes, you can give him a badge.”
“Here you go little girl.”
“Aaaaaaaaaaaaarg!”
(Not to mention that one staff member not only though The Younger was a girl, but still wore diapers! I mean, yes, we're waiting for an event from down there, but The Younger can hit the bathroom for that. Sheesh!)
Didn't we learn that long hair isn't an indication of gender during the 70s?
It's enough to make me want to put my head in my hands and sob uncontrollably.
gemini://gemini.conman.org/boston/2006/01/16.2
On the way to work (after dropping off Wlofie and The Older at The Hospital [1]), I stopped off at a gas station to fill up Lake Lumina. There are two pumps next to each other, both free. I'm trying to negotiate my way there when an SUV (Sport Utility Vehicle) cuts in. But does the soccer mom driving said SUV pull all the way forward?
No, of course not.
She pulls up to the first pump, and stops.
Would it hurt her to pull up to the next pump, free of any vehicular obstruction and fill up there?
Yes, of course it would.
So I have to drive around this bree munching Yuppie soccer mom and circle around to some other free pump.
Then, when said bree munching Yuppie soccer mom finished filling up her SUV (I didn't actually see her get out of the vehicle and pump any gas, although I did see her talking on a cell phone), does she pull forward to circle around the lot to leave?
No, of course not.
Instead, she backs up and tries to negotiate a three point turn right there in front of the pump.
It's at this point that I want to put her head in my hands and shake violently.
gemini://gemini.conman.org/boston/2006/01/16.3
**From:** Dan the network engineer <XXXXXXXXXXXXXXXXXX> > **To:** Sean Conner <sean@conman.org> > **Subject:** > **Date:** Mon, 16 Jan 2006 11:12:05 -0500
Hey Sean,
XXXXXXXXX's traffic issues [1] have not been solved, and, in fact, are still causing some major issues.
I believe this server has been hacked and needs to be taken off-line …
Okay, see! **This** is a Monday [2] thang …
Sigh.
gemini://gemini.conman.org/boston/2006/01/16.4
Well, the server was hacked [1], but it looks to be a customer account was compromised, since the executables where owned by a customer account, the processes were running on unpriviledged ports, and the server was being used as part of denial of service attacks, with executables hidden under a hidden directory in /var/tmp.
Fortunately, the system hacked is running Linux without module support, so patching system calls [2] to hide activity is impossible without a reboot (which would be noticed).
And as always, it could have been worse [3].
[2] http://lib.ru/SECURITY/linux_module_heroin.txt
gemini://gemini.conman.org/boston/2006/01/17.1
The magnets are still stuck in The Younger [1]. And this morning he was in pain. Quite a bit of it too. Hopefully the pain means they are on the move, but we won't know until later today after his insides are scanned once again [2].
If the magnets haven't moved, tomorrow the doctors go in.
Here's hoping the magnets leave his system on their own.
[2] http://www.betterhealth.vic.gov.au/bhcv2/bhcarticles.nsf/pages/Barium_meal?OpenDocument
gemini://gemini.conman.org/boston/2006/01/17.2
I finished writing the real-time LaBrea data processing program [1] (most of it last night at The Hospital) and the final few bugs were real doozies.
The program works by getting data from LaBrea [2], then it looks up the connection and updates the information accordingly. The code pretty much looks like:
>
```
void start_tarpit(time_t stamp,char *line)
{
struct tprecord *exists;
struct tprecord rec;
size_t index;
read_record(&rec,line);
exists = tpr_search(&rec,&index);
if (exists == NULL)
exists = pull_free_record();
exists->src = rec.src;
exists->sport = rec.sport;
/* ... */
record_add(index,exists);
}
/*******************************/
struct tprecord *pull_free_record(void)
{
if (g_poolnum == g_poolmax)
do_forced_garbage_collection();
return(&g_pool[g_poolnum++]);
}
/*****************************/
void record_add(size_t index,struct tprecord *rec)
{
if (g_recnum == g_recmax)
do_forced_garbage_collection();
memmove(
&g_rec[index + 1],
&g_rec[index],
(g_recnum - index + 1) * sizeof(struct tprecord *)
);
g_rec[index] = rec;
g_recnum++;
}
```
tpr_search() is the binary search routine I wrote the other day [3], and as you can see, it returns the index to where the record is in the array, or to where it should be. pull_free_record() just returns the next free slot in the structure array, and if there are no slots available, it does a removes some older records according to some criteria. And record_add will add the record to the pointer array, also removing older records if there is no space left.
Some records are deleted. All remaining records move about. The pointer array is resorted. So between the time
>
```
exists = tpr_search(&rec,&index);
```
and
>
```
record_add(index,exists);
```
index may not be a valid index anymore!
Oops.
(Never mind the fact that one of the two calls to do_forced_garbage_collection() is redundant)
Simple enough to fix once I knew what was going on.
Another bug dealt with named pipes. Instead of directly piping the data from LaBrea to ltpstat (what I call the read-time LaBrea data processing program), I decided to go through a named pipe, which would allow me to start and stop either one independantly from the other.
Now, I'm testing my program, running cat sample > labrea.pipe in one window, and ltpstat --input labrea.pipe in another. It's working, until the data in sample runs out and cat closes its side of the named pipe.
Now, the code that reads in the data is in a library I wrote, and it just assumes when a read() returns 0, that it's the end of the file, and marks it as being closed. ltpstat ignores the “end of file” status, and keeps trying to read a now-closed file. We get into a busy loop and the system load shoots up. Also, if I now try to pump more data through the named pipe, ltpstat ignores the data.
Even when I modify the library code to not mark “end of file” when there's nothing to read does nothing, as from that point on, read() just returns nothing anyway and seems to be a “feature” of Linux (or Unix—I didn't have Advanced Programming in the Unix® Environment [4] with me to look this up), so I restructure the main loop:
>
```
while(1)
{
in = openinput();
while(!StreamEOF(in))
{
process_labrea_output();
}
StreamFree(in);
}
```
That was all fine and good, until I threw signals into the mix.
I use signals to tell ltpstat (what I call the read-time LaBrea data processing program) to dump various information—SIGHUP to print the number of connections and unique IPs being tracked, SIGUSR1 to do a raw dump of all the data accumulated so far and SIGUSR2 to generate a more or less human readable dump.
But signals are basically interrupts. And operating system theory states that one process should never find another process with its pants down (PCLSRing: Keeping Process State Modular) [5] (so to speak—and I should warn you—the paper I linked to is way more technical than I've gotten here). Signals and system calls interact (that is, if a process is signaled while making a system call into the kernel) in one of two ways—the system call will simply fail, or it will be restarted automatically. And one can select which method to use.
If I elected to have the system call fail, and ltpstat would fail, either to open (a system call) the named pipe, or in reading (another system call) the named pipe after I signal the program.
If I elected to have the system call restarted, and the signal handlers I set up would never get called (due to the way I handle the signals).
I ended up reimplementing two routines from my library (which is used in more than just this program) for just this program. I select for the “system call fail” method, check to see if the system called failed due to a signal, and if so, check for the signals, and try again.
Again, this took a few hours to track down.
But now the program works, and I can finally get real time statistics from LaBrea.
[2] http://sourceforge.net/projects/labrea
[4] http://www.amazon.com/exec/obidos/ASIN/0201563177/conmanlaborat-20
[5] http://fare.tunes.org/tmp/emergent/pclsr.htm
gemini://gemini.conman.org/boston/2006/01/17.3
I just received a call from Spring [1]—the mangets are in The Younger's stomach! (why this wasn't found out sooner, I don't know). So, because it's in his stomach, he's being moved yet again to another hospital, one that has the equipment that might get the magnets from his stomach.
I'll be heading there once I get word which hospital it is.
gemini://gemini.conman.org/boston/2006/01/17.4
While I'm waiting a call back [1], some more on LaBrea [2].
Yesterday (from January 16 at 06:28:25 to January 17 08:54:50) LaBrea generated 1.1G (gigabytes) of log data, and it took full five minutes to run grep 'Initial Connect' daemon.log.0 | wc -l (255,344 new tarpitting connections by the way).
LaBrea was also running at full speed, maxed out at 64Kbps (Kilobits per second) bandwidth to keep all these connections tarpitted (the maximum I set LaBrea to use, by the way).
[NETWORK TARPIT bandwidth usage] [3]
That first large dip in the graph (the one around 6:30 in the morning) is probably due to the system attempting to rotate a 1.1G log file. The second dip, at the right (around 3:00 pm) is when I restarted LaBrea so its logging information would go through ltpstat [4]. After an hour of running:
>
```
Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 15:55:59 2006 Running time: 1h
Pool-max: 1048576
Pool-num: 24322
Rec-max: 1048576
Rec-num: 24322
UIP-max: 1048576
UIP-num: 1282
Reported-bandwidth: 40 (Kb/sec)
```
And after two hours:
>
```
Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 16:56:19 2006 Running time: 2h 20s
Pool-max: 1048576
Pool-num: 33326
Rec-max: 1048576
Rec-num: 33326
UIP-max: 1048576
UIP-num: 1632
Reported-bandwidth: 40 (Kb/sec)
```
And right this second:
>
```
Start: Tue Jan 17 14:55:59 2006 End: Tue Jan 17 18:37:19 2006 Running time: 3h 41m 20s
Pool-max: 1048576
Pool-num: 42931
Rec-max: 1048576
Rec-num: 42931
UIP-max: 1048576
UIP-num: 2148
Reported-bandwidth: 40 (Kb/sec)
```
Okay, pool-max and rec-max are the maximum sizes for the structure array and pointer array, and both should always be equal at all times (I'm displaying this number more for debugging purposes than anything else), while pool-num and rec-num (which should also be equal at all times) represent the current number of connectioned tarpitted. I also keep track of unique IP (Internet Protocol)s, which just now is currently 2,148 (out of 1,048,576 that I can store). I also just found out that IP address 195.130.152.85 has 4,809 connections currently tarpitted (and in the few seconds it took to do that query, five more connections were tarpitted).
I'll be releasing this code in the next few days, when I can write up some documentation and slap on a license.
[2] http://sourceforge.net/projects/labrea
[3] /boston/2006/01/17/tarpit.20060117.png
gemini://gemini.conman.org/boston/2006/01/17.5
Just now received a call from Spring [1]—the trip to the other hospital [2] has been postponed until tomorrow morning.
That is all.
gemini://gemini.conman.org/boston/2006/01/18.1
The Younger is waiting to be transported to the next hospital in line [1]. The magnets are still in his stomach waiting to be removed [2].
So, hurry up and wait …
gemini://gemini.conman.org/boston/2006/01/18.2
Just because I was curious, I ran the Perl script that parses the LaBrea logs on the 1.1G (Gigabyte) logfile from the other day [1]. I knew it would take at least four hours or so [2] to run.
I then created a modifed version of ltpstat to parse this file (it's the output from syslogd, hense some slight parsing modifications were required) and ran it at the same time.
What I found interesting (during the actual run of both programs) is that my program had a larger virtual memory footprint over the Perl version (easily seven to eight times larger) but the resident set size (the amount of physical memory being used—the rest not physically allocated or shoved off into swap space) of my program was half that of the Perl script. In retrospect this was expected—Perl was growing the data as it was being generated whereas my program allocates the whole thing at once, but my program has less overhead in keeping track of said data.
And ltpstat is faster than the Perl script even if it isn't gathering the stats in real-time— 3 hours, 22 minutes and 24 seconds to run vs. 6 hours, 39 minutes 49 seconds—almost half the time. I didn't see how much memory the Perl script was using just prior to finishing, but I can't see how it would be less than ltpstat.
The instance of ltpstat I started yesterday [3] is still running:
>
```
Start: Tue Jan 17 14:55:59 2006 End: Wed Jan 18 14:55:59 2006 Running time: 1d
Pool-max: 1048576
Pool-num: 388929
Rec-max: 1048576
Rec-num: 388929
UIP-max: 1048576
UIP-num: 20298
Reported-bandwidth: 64 (Kb/sec)
```
Looks like I may break a million connections sooner than I expected.
gemini://gemini.conman.org/boston/2006/01/19.1
Latest update on The Saga of the Younger [1]: he's been moved to the next hospital in line, and there the doctors discovered that the magnets are stuck in an ulcer in his stomach, and the only way to get them out is surgery. We have no time frame for when said surgery will happen though.
Updates as I get them.
gemini://gemini.conman.org/boston/2006/01/20.1
The Younger is out of surgery [1]. That is all I know at this time.
The magnets did get separated—one ended up in his stomach and the other one ended up in his intestines. They did attract each other and the attraction was enough to cause them to borrow through tissue.
They've been removed (and were quite corroded from what I understand) and the holes sewn up. He's in recovery for the forseeable future. So now it's just waiting around until he heals up.
gemini://gemini.conman.org/boston/2006/01/21.1
I have internet access for the evening, courtesy of having to come to work. Trying to catch up on the most essential emails and such, in between working cases.
I love you people. Thank you so so so much for the messages of concern, wishes, prayers, and other good vibes, as well as information on polar bear blubber and vitamin poisoning. Thanks also for offers of assistance. We've got things covered, but truly appreciate it.
## Magnet news:
On the 19th the team at West Boca sent an endoscope down his gullet, under general anaesthesia. They found an ulcer with a shadowy something in the middle that was likely the magnets. That something in the middle couldn't be gotten out. Surgery required.
The surgery occurred yesterday, the 20th, and took longer than anticipated—it was about 2 hours. rhonan [1] had it right—one magnet had gone down the small intestine, and the other had stayed in the stomach, and they attracted. The ensuing irritation and vomiting episode caused them to begin tunneling toward each other, scar tissue forming a tube. Surgeons had to disconnect the tube, remove the magnets, and fix the holes. They were thrilled at how fixable it was, though - no ancillary damage like necrosis or sepsis or any of that. It could have been far worse.
Afterward, straight into ICU (Intensive Care Unit) and morphine. It's not fully effective against the pain, though. He got out of ICU today and is still convalescing, no food or liquid yet. The innards have to heal first.
My weekend is starting after tonight's shift, and I'll most likely be spending all of it at the hospital. I'll certainly be there after shift tonight. Will continue to post phone-to-email-to-LJ updates as necessary and able.
“Spring Dew: A quick game of ketchup [2]”
Not much more to add to this (except I don't have a phone-to-email gateway to update this site).
[1] http://rhonan.livejournal.com/
[2] http://springdew.livejournal.com/397627.html
gemini://gemini.conman.org/boston/2006/01/21.2
You would not believe how hard it was to write a binary search that returned the correct index for a missing record in an array.
“Some notes on a binary search implementation [1]”
A week later, and I finally have it working.
One technique used to debug a program is to have another program that does the same thing, but implemented using a different method or language or both. And I did. I ran the Perl program I had over the 1.1G (Gigabyte) log file [2], then ran ltpstat over the same log file and got two different results.
Not good.
ltpstat returned 2% more connections than the Perl script. Getting a dump from the currently running version on the LaBrea [3] system and cleaning the output showed a 2% difference again.
So I spent the past week trying to track down the problem. It was obvious that ltpstat was storing duplicate records, but why was a different matter. My testing sample of about 1,100 connections is apparently too small to completely test the program, so I had to test using the 1.1G log file which has approximately 230,000 connections.
To help debug this problem, I wrote a linear search and would call it as well as the binary search. If both agreed, then I would return the information, otherwise, I would log the discrepency, do the search again, then exit. The reason for doing the search a second time? So I could set a breakpoint there, and let the program run for a couple of hours until it triggered. Then I could step through both searches to see where the problem was.
Yup, each run took several hours to trigger the bug.
I ended up testing four different binary search routines (including the original one I thought worked, plus one I modified from The Standard C Library [4], plus two other versions I wrote) before sitting down and working through things on paper.
And I still missed corner cases.
But finally, I tested my final version it against the Perl script and only had 122 discrepencies out of some 230,000 records (or 5% of 5%—too small for me to worry about after spending a week on this).
* * * * *
I took a snapshot of the currently running version (which had been running for a bit over three days now), cleansed the output of duplicates, and the final tally was 416,230 connections from 12,911 unique IPs. Again, nothing surprising about the ports being attacked:
Table: Top 10 ports captured by Labrea in the past 3 days Port # Port description # connections ------------------------------ 139 NetBIOS (Basic Input/Output System) Session Service 160,799 135 Microsoft-RPC (Remote Procedure Call) service 108,958 445 Microsoft-DS (Directory Service?) Service 67,506 80 Hypertext Transfer Protocol 23,921 4899 Remote Administration [5] 9,225 22 Secure Shell Login 7,253 1433 Microsoft SQL (Standard Query Language) Server 6,503 8080 Hypertext Transfer Protocol—typical alternative port 3,717 3128 Squid HTTP (HyperText Transport Protocol) Proxy 3,329 1080 W32.Mydoom.F@mm worm [6] 3,150 ------------------------------ Port # Port description # connections
And again, the Microsoft specific ports account for 81% of the scans. I'll need to discuss with Smirk about blocking those ports in the core router. If nothing else, LaBrea is giving me an indication of which ports to block.
[3] http://sourceforge.net/projects/labrea
[4] http://www.amazon.com/exec/obidos/ASIN/0131315099/conmanlaborat-20
[6] http://securityresponse.symantec.com/avcenter/venc/data/w32.mydoom.f@mm.html
gemini://gemini.conman.org/boston/2006/01/22.1
It's weird. I have my email address sean@conman.org [1] plastered all over my website here, and yet, I'm receiving a bazillion spams for a type of sexual aid at my private email address. But at my oh so public email address [2]?
Nothing what so ever about this particular sexual aid.
How odd.
gemini://gemini.conman.org/boston/2006/01/23.1
I remember having this weird dream about ten years ago or so. In this dream, my boss (it wasn't you Smirk) walked up to me, opened his mouth and said “Waaaaah waaaaah waaaaah waaaaah waaaaah.”
“What?” I said.
“Waaaaah waaaaah waaaaah waaaaah,” said my boss. He then went around the office continuously saying “waaaaah waaaah waaaah,” even if he wasn't speaking to anyone in particular.
I do recall in my dream that a bunch of us were trying various methods to shut him up, since we all found his “waaaaah waaaah waaaaah” very annoying. After what seemed like hours, it dawned on me that this wasn't the normal behavior of our boss and that something else might be to blame. It was then I realized I was dreaming, and woke up. To my alarm clock, which was going “waaaah waaaaah waaaah waaaaah waaaaah” and had been for the past 40 minutes.
Yes, I'm a heavy sleeper.
I also found out (about the same number of years ago) that my alarm clock would simply give up after three hours of me hitting the snooze button (not only did I hit the snooze button for three hours, the alarm clock itself was on the other side of the room—yes, for three hours, every nine minutes, I would roll out of bed, take two steps across the room, hit the snooze button, two steps back and roll back into bed, without disturbing my sleep!)
I no longer use that alarm clock—oh, it still works, it's just that the hour button has broken off (personally, I would have expected the minute button to break off) so it's rather difficult to set. I've been using my cell phone as an alarm clock. No snooze mode, so I have to be semi-conscience to manually set the alarm, but the alarm itself only sounds for a minute, which may be too short a duration.
I only mention this to explain to Smirk was I was nearly three hours late to work today.
gemini://gemini.conman.org/boston/2006/01/24.1
The Younger started in on liquids today, and it looks like he'll be eating solid food starting tomorrow. That, and he's flirting with the nurses, so he's healing right up. I'm expecting he'll be in The Hospital for the next few days or so.
gemini://gemini.conman.org/boston/2006/01/24.2
I have to wonder the thought process of a script kiddie that runs the SSH22 Scanner (which tries to log into servers via ssh using a series of precanned userids and passwords) as fast as it can. Doing so only brings attention to the fact that it's sucking up network bandwidth and loading down the box, gaining the attention of the sysadmin, thus negating the purpose of running such a program in the first place.
I also have to wonder at the thought process of a script kiddie that logs back into a box after loosing the connection and not bothering to see if anyone is logged in as root before starting the SSH22 Scanner back up.
gemini://gemini.conman.org/boston/2006/01/25.1
> > If someone was to drop a bomb on this building, it would wipe out 50 percent of the Lisp community. That would probably be a good thing. It would allow Lisp to start over.
>
That wasn't exactly what John said, but it was pretty close. Again, the main thing here was somebody with impeccable Lisp credentials saying that Lisp was far from perfect and in fact needed to do better than Common Lisp. The fact that Patrick, Henry, and John all argued essentially the same thing was a wake-up call, I think.
“Finding Lisp: ILC 2005, Wednesday report (late) [1]”
Now, I have this love-hate relationship with Lisp. I love the idea of Lisp, and it is a cool language, but I hate the implementations of it (it doesn't play well with other languages), lack of syntax (which expresses itself in Lisp as a metric buttload of parenthesises), the nasty low level details one has to keep in mind (really! In Common Lisp [2], 2 (this particular instance of the number) is not necessarily equal to 2 (this other instance of the number)—I kid you not!) and the rather archaic way it still operates (CAR and CDR are remnents from the very first implementation where one took the “contents of the address register” for the current element in a list, and “contents of the decrement register” for the rest of the list—even though you now have FIRST and [DELETED-REST-DELETED] [DELETED-LAST (oh, it's called LAST even though it returns the rest of the list)-DELETED] REST (apparently, I had it right the first time, and LAST is used for something else in Lisp—sigh) nobody ever uses those).
Oh, and to get any real speed out of Lisp you have to forego using lists (which is what Lisp stands for—“LISt Processing”) and use more specific data structures (never mind the fact that Lisp is still taught as if it only has lists).
The “John” mentioned above is John McCarthy [3], inventor of Lisp, which is one of the two oldest programming langauges (the other being FORTRAN (Formula Translator)), and if he, of all people, doesn't have much positive to say about Lisp, then what chance does it have of gaining wide spread acceptance in the industry (Greenspun's Tenth Rule [4] notwithstanding)?
[1] http://www.findinglisp.com/blog/2005/06/ilc-2005-wednesday-report-late.html
[2] http://www.apl.jhu.edu/~hall/lisp.html
[3] http://www-formal.stanford.edu/jmc/
[4] http://en.wikipedia.org/wiki/Greenspun's_Tenth_Rule
gemini://gemini.conman.org/boston/2006/01/26.1
The Younger has left the hospital!
Thank God that's over.
And if The Younger ever eats another magnet again, I'll kill him! Sheesh!
![[2] /boston/2006/01/11/thumb.emergency.jpg](/gemini.conman.org/boston/2006/01/11/thumb.emergency.jpg){kind=link}
![[3] /boston/2006/01/11/emergency.jpg](/gemini.conman.org/boston/2006/01/11/emergency.jpg){kind=link}
![[3] /boston/2006/01/17/tarpit.20060117.png](/gemini.conman.org/boston/2006/01/17/tarpit.20060117.png){kind=link}