💾 Archived View for gmn.clttr.info › sources › geminispace.git › tree › docs › handling-robots.md.tx… captured on 2022-01-08 at 21:32:56.

View Raw

More Information

⬅️ Previous capture (2021-12-03)

-=-=-=-=-=-=-

# robots.txt handling

robots.txt is fetched for each (sub)domain before actually crawling the content.

GUS honors the following User-agents:

	
indexer
	
*


## robots.txt caching

Every fetched robots.txt is cached only for the current crawl.