I'm sorry, I just find this stuff interesting.
Marcus [1] wrote in, asking why Google [2] seems to have abandoned his weblog, although I'm not sure if he was asking about referers from Google, or Google actually spidering [3] his site. Hard to say what is going on, and from doing some research on my own server I did come across some rather interesting figures.
Table: Google spidering vs. Google referers for The Boston Diaries Date of log Googlebot hits Google referers Comments ------------------------------ October 2001 0 0 blog went live 10/22 November 2001 721 26 December 2001 1,421 348 January 2002 579 733 February 2002 341 1,073 something wrong with Googlebot? March 2002 854 694 April 2002 1,019 649 May 2002 1,073 675 June 2002 1,228 504 July 2002 994 512 incomplete month
While traffic from Google's spider has gone up, actual traffic from their engine has gone down. I'm not sure if they've tweaked their page ranking algorithm to decrease the their sensativity to blogs, or I'm now fighting with a bazillion other weblogs for Google traffic, or now that I've been live for nine months things have settled down and I can expect a similar level of traffic here from now on. But, for contrast, I decided to scan a static (relatively unchanging) website—www.conman.org (for the same time period as my online journal/weblog):
Table: Google spidering vs. Google referers for Conman Laboratories Date of log Googlebot hits Google referers Comments ------------------------------ October 2001 185 286 blog went live 10/22 November 2001 275 301 December 2001 435 395 January 2002 279 370 February 2002 74 462 something wrong with Googlebot? March 2002 226 393 April 2002 226 307 May 2002 382 358 June 2002 540 293 July 2002 215 285 incomplete month
You can see here that the level of traffic from Google searches is pretty constant, although the Googlebot seems to go up and down. It is apparent though, that something was up with Googlebot in February as visits from it dropped dramatically for that month (and I'm only saying for those two months—I haven't actually checked any of the other sites I host).
And after all this checking, I still don't have a difinitive answer for Marcus as to why Google has slacked off his site. My own experience is that Google likes conman.org—part of that might be the stableness of the URLs on all parts of conman.org (my personal site, www.conman.org/people/spc/ [4] used to be hosted elsewhere, but when I first moved it to conman.org [5] in October of 98, I placed permanent redirects from the old site to conman.org so the engines at that time immediately found my new site) and there's something to be said for stability. It might also like my online journal/weblog (but Google seems to have an afinity for those in general). I'm not sure what to tell Marcus though.
[1] http://viasteallarum.antville.org/
[3] http://www.googlebot.com/bot.html