2009-06-12 Referrers

I wrote a little Perl script that parses Apache log files and counts referrers. It’s pretty specific for this site, since it tries to determine for both site and referrer whether the log entry is about Emacs Wiki, my blog, Oddmuse, etc.

Emacs Wiki

Oddmuse

The numbers for my blog in yesterday’s 24h period: 75307, which is 5% of the total number of hits in the log file. Of these, 50276 hits had no referrer information, 13196 came from my own site.

Now, to understand this, one needs to understand that most of the traffic on the web is caused by search engines. At least for my site!

aschroeder@thinkmo:~$ egrep 'GET /(cgi-bin/)?alex' logs/access.log.1 | wc -l
75308
aschroeder@thinkmo:~$ egrep 'GET /(cgi-bin/)?alex.*google' logs/access.log.1 | wc -l
18202

Fully 24% of all my traffic is Google related!

But now, some numbers: I still have 75307-50276-13196=11835 hits to explain! 😄

rpgbloggers.com: 5962 – RPG Bloggers results in a lot of hits. That’s pretty amazing. Not many people comment. Either my articles are so perfect (haha), or so full of it (although that has never stopped trolls from commenting elsewhere), or just plain not interesting. I guess I rarely hit the mainstream when it comes to RPG blogging.
emacswiki.org: 2329 – Emacs Wiki remains my best known project and has a lot of visitors. As a link to my homepage can be found on my profile, I guess that’s how they find it. But I haven’t checked this.
google: 888 – various Google sites from all over the world. I’m surprised that so many people find me via Google. I’ll have to check whehter these entries are from the Google Bot or from actual people. If so, I’d also learn something about the queries used to find the blog.
odd74.proboards.com: 567 – this must be based on the small number of messages I posted on the Fight On Forum.
communitywiki.org: 467 – Community Wiki is a dormant project of mine. Still a lot of referrals!
jrients.blogspot.com: 147 – Jeff Rients’ blog? I left a few comments there, and it is in fact a very cool blog. I didn’t know that so many people clicked through to my blog. Here, too, it would be worth while trying to figure out whether there is one comment in particular that’s at the source of this.
chattydm.net: 122 – Chatty DM is a blog that I rarely comment on. I guess the reason for the current referrals is my participation in the One Page Dungeon Contest 2009; strictly speaking that would be yet another hypothesis to test, however.
tiddlywiki.com: 134 – I haven’t used TiddlyWiki in ages, but years ago I wrote a little hypertext dream using it. I think it got linked from the main site, so that would explain the big number.
poietic-aggregator.com: 63 – I think the Poietic Aggregator is a project by some wiki people. It’s a weird RSS visualizer. I never got the hang of it.
orientalisch.info: 60 – Claudia’s website. I’m surprised that so many people click through. This must be something else. I should investigate this.
blo.gs: 59 – I’m not sure whether I’m still pinging the site. Right now the site seems to be inbetween owners. Always a bad sign. I wonder how they manage to refer to my blog anyway.
finch.ploogy.net: 54 – the first *real* surprise. A website that “makes slow Internet bearable, by stripping away the fat of web pages, leaving just the content. It takes out CSS, images, flash, metadata, iframes and more, meaning less for your computer to load.” What a cool idea. I hope that Finch doesn’t really find much stuff to strip from my web pages. When I check them with YSlow I get an A grade for everything but “use a content delivery network” (not big enough) and “compress components with gzip” (I thought about this but this requires me to know the size of the content before serving headers, and I prefer to server as quickly as possible).
campaignwiki.org: 52 – not many people are using Campaign Wiki, a project of mine offering a very simple wiki without ads for gamers.
bloglines.com: 44 – a blog aggregator like google reader. Interesting.
wandersite.ch: 39 – a hike or two of mine are linked from that site.
rsp-blogs.de: 35 – the German version of rpgbloggers.com!
bing.com: 35 – search engine
oddmuse: 23 – another software project of mine
bitethebulette.blogspot.com: 22 – a enw blog that I’ve discovered recently

my participation in the One Page Dungeon Contest 2009

And many other small hits.

#Blogs

Comments

(Please contact me if you want to remove your comment.)

⁂

I am probably responsible for some of the Bloglines.com stuff, as I use that particular feed aggregator.

– Adrian 2009-06-14 19:24 UTC

---

Ah! 😄 I switched from Bloglines to Google Reader a few weeks ago and was surprised other people still use it.

– Alex Schroeder 2009-06-14 22:26 UTC

Alex Schroeder

---

Yeah, I am slow with the technology. Google Reader does make some sense, though, since I am already using Gmail and Google Calendar...

– Adrian 2009-06-16 13:19 UTC

---

This is interesting. Thanks.

I’m surprised how much syndication directs traffic. I actually thought search engines would be the big draw to your site. Although this confirms that if you write a good blog and even if it has the syndication technology – a feed – doesn’t mean the visitors will find you. You can only get traffic if you advertise your blog by adding it to to a collective feed (an aggregator called a “planet”) or by commenting on other blogs.

– AaronHawley 2009-06-17 16:24 UTC

AaronHawley

---

Plus it makes me wonder whether 25% of all the hits and more for search engines is worth the price since I’m getting relatively few referrers. One would have to figure out a “conversion rate” for visitors from Google. My guess that I should introduce some sort of rate-limit for bots. What do you think?

– Alex Schroeder 2009-06-17 22:24 UTC

Alex Schroeder

---

I am always spellbound how little time it takes Google to find a new page and have it show up in search results. In my mind, I imagine that Google’s bot cluster is crawling all of the Web all of the time. With the development of blogging and their news media interface, I predict they are constantly pinging pages to find changes and new pages.

Yeah, if only you could rate-limit a bot with the ratio of traffic their search engine brings you? 😄

– AaronHawley 2009-06-18 17:36 UTC

AaronHawley