My Search Engine has now indexed more pages than Kennedy and TLGS, and has crawled more domains than Geminispace.info! :)
gemini://auragem.letz.dev/search/
5 months ago · 👍 willowf, maxheadroom, docedelgasforgot2renewcert
gemini://auragem.letz.dev/search/
@remy Thanks for informing me of how many pages that site has. I try not to get involved in what the crawler indexes, because I want it to be representative of as much of geminispace as possible, but I do sometimes make exceptions. I will think about it.
Also, the search engine doesn't actually store all of the content of a site, it only stores headings and link text, iirc. · 5 months ago
Your crawler is indexing gemini:/gmi.noulin.net/stackoverflow/
It has 3.2 million pages (18GB). · 5 months ago