💾 Archived View for bbs.geminispace.org › u › Acidus › 13264 captured on 2024-02-05 at 12:17:46. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Re: "Any experience with data scraping / crawler here on GEMINI?..."
Many crawlers in geminispace don't response robots.txt. They don't even request it. I constantly have naive crawlers endlessly crawling all of Wikpedia via my Gemipedia CGI, or endlessly crawling news websites via my NewWaffle CGI. It's rather annoying and when I see it I block those IP's with iptables.
One thing I do, that I wish other crawlers did, is request robots.txt with a query string telling what the crawler is. For example, Kennedy always send a request for /robots.txt?kennedy-crawler before requesting any other URLs
2023-12-30 · 5 weeks ago
Any experience with data scraping / crawler here on GEMINI? Can it be verified? I became aware of DARK VISITORS and wonder if it is worth implementing something from there. [https link] ( A list of known AI agents on the internet)