💾 Archived View for ew.srht.site › en › 2020 › 20201223-re-crawling.gmi captured on 2023-04-19 at 22:59:15. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2022-03-01)
➡️ Next capture (2024-06-16)
-=-=-=-=-=-=-
Alex Schröder wrote:
What would a network look like that doesn't crawl?
People say: that’s the only way a search engine can work. Maybe? Maybe not? What if sites sent updates, compiled databases? A bit like the Sitemap format? A sort of compiled and compressed word/URI index? And if then very few people actually sent in those indexes, would that not be a statement in itself? Now people don’t object because it takes effort. But perhaps they wouldn’t opt-in either!
gemini://alexschroeder.ch/page/2020-12-22_Crawling
Reading this struck some chords.
It is /not/ an unavoidable trait of this physical universe that there be crawlers --- possibly known as "nosy noisy hungry bots" :) --- Maybe they have to be noisy on the net, because they must somehow increase entropy in the universe :) I start to see the jungle --- yet I know, I would be killed on the very first contact in a real jungle.
I have contemplated the idea of buying some small virtual machine instance in a datacenter. But so far I have resisted, because I am not interested to guard this smol machine. It is a time sink on details I consider not interesting enough. And it seems to me, I'm not alone with this.
Thanks for writing this up!
Cheers,
~ew