I came across a micropayment scheme that is making the rounds: Penny per Page [1] and it works just like it sounds—you pay one penny to view one page. Technically, it's possible. HTTP (HyperText Transport Protocol) has provisions to expand for pay-for-reference (although no standard is mentioned) and some work (W3: Micropayments Overview) [2] has been done.
The fact that they don't pay for Web content is a historic anomaly. The benefits to be reaped by paying a very small amount of money for Web content are gigantic. Right now, people are actively denying themselves many of the most amazing things that the Web could provide because of the "totally free" World Wide Web.
How Penny Per Page Might Work page 4 [3]
The article [4] even mentions how under this scheme, Google [5] could easily make $350 million a year (assuming Google can maintain it's 100 million page hits per day) but see—there's a slight problem and it's a problem I haven't seen mentioned in any of the micropayment schemes I've read up on: search engines.
Ah yes, the Google Problem (as I've come to call it). The whole point of a search engine is to catalog your site so others can find it. If no one can find your site, it doesn't matter if you charge 1¢ or $1—you're not going to make money. And generally, sites don't mind if a search engine crawls through the site and indexes it. Heck, there are companies that make money submitting sites to search engines so they'll be crawled.
Now, how much of that fabled $350 million that Google [6] makes will stay if Google has to pony up the 1¢ for each page it fetches?
Now, statistically speaking, using only my site [7] and extrapolating from there makes poor science but hey, it's a starting point. A quick scan through the logs (of www.conman.org [8], bible.conman.org [9], literature.conman.org [10] and boston.conman.org [11]) which so far only covers November 1^st through the very early morning hours of the 16^th (it's 3:08 am as I'm writing this) I've had 986 visits from Googlebot [12] but only 83 referals from Google [13] itself.
Interesting! Under this hypothetical plan, Google lost $9.03 on spidering my site. If I check all the sites I host, Google lost $15.46 from all the spidering it did. Meanwhile, I made $10.69 from Google spidering just conman.org or if I consider all the sites: $22.54.
On a whim, I checked three other sites whose logs files I have access to to see if the rather ad-hoc theory I'm working under is valid. Two sites Google paid more to visit than they made in search results, but definitely came out ahead on the third (of course it's a sex-related site).
So it would be hard to say if Google would be able to keep the $350 million if it too was subject to paying out 1¢ per page it indexed.
The other side of the coin is for the search engines to be exempt from the penny-per-page charge—after all, they're driving visitors to the site after all. But then it becomes a problem of determining if what is going through the pages is a robot or not. If you base the decision on the User-Agent then what's to stop someone using Opera [14] and changing its User-Agent string to say it's Googlebot? Authentication is one method, but it's hard enough getting robots.txt [15] on all sites and that's a simple text file. Something as complicated as an anthentication scheme for robots is going to be tougher to sell.
[1] http://www.howstuffworks.com/penny-per-page.htm
[2] http://www.w3.org/ECommerce/Micropayments/
[3] http://www.w3.org/ECommerce/Micropayments/
[4] http://www.w3.org/ECommerce/Micropayments/
[10] http://literature.conman.org/
[11] http://boston.conman.org/
[12] http://www.googlecot.com/bot.html