📅 Published 2021-09-12
Conventional wisdom has it that you should immediately submit your new blog or website to all the major search engines. On top of that, you should install some kind of "analytics" tracker so you can learn as much as possible about your "audience" and what it is they want.
Screw the conventional wisdom.
On the WWW version of this site, my `robots.txt` file now looks like this:
User-agent: WibyBot User-agent: search.marginalia.nu Disallow: User-agent: * Disallow: / Sitemap: https://mntn.xyz/sitemap.xml
I like Wiby and marginalia's project, so they can index whatever they want. As for the rest, they can buzz off! (If you know of another "smol web" search engine I should add to my exclusions, please send me an email.)
I have no robots.txt file on the Gemini site, as the only known Gemini search engines are fine by me.
Why did I do this? Well, first of all, this is a personal project, and I'm not out to make money on it. I figure that the handful of people who would be interested in this site will find it somehow, probably through Gemini. I may eventually join a webring; we'll see.
Secondly, I don't particularly *want* the masses flocking in to my small blog, gawking about, tracking mud on the rug. It's cozy in here. There's no reason to bring in the whole world! I'm not eager to hear the perspective of a hundred armchair critics.
Mostly, though, I'm tired of the influence that the search giants have on the web. I can't tell you how many times I've searched for something that I **know** exists, but no search engine will show it to me. Each will claim "40 million results" and then quit showing me those results after page 5. Or I'll search for something relatively common and get zero results. I can't believe I am saying this, but I miss the days of Altavista, where binary operators *worked* and you had to learn to sift out spam on your own. I was really good at it, and I could find almost anything! These days "advanced" search operators are more of a suggestion; engines often ignore them entirely.
(Yes, I would pay money for a private, "filter-less" search engine. Perhaps it could have optional open source filters, developed like UBlock/AdGuard filters.)
Anyway, I have no desire to feed more data and therefore more traffic to these behemoths. Let the Eye of Sauron gaze somewhere else.
---
Comments? Email the author: mntn at mntn.xyz