#! # This is the robots.txt file for the Gopher and Gemini version of this site. # See gopher://alexschroeder.ch/02019-05-24_Robots_and_Gopher # or gemini://alexschroeder.ch/2019-05-24_Robots_and_Gopher # The user-agent line is required for Gemini and illegal for Gopher. Damn! User-agent: * Disallow: */history Disallow: */append Disallow: */write Disallow: */html Disallow: */tag Disallow: tag/* Disallow: raw/* Disallow: history/* Disallow: diff/* Disallow: html/* Disallow: map/ Disallow: do/rc Disallow: do/rss Disallow: do/new Disallow: do/more Disallow: do/tags Disallow: do/match Disallow: do/search # allowing do/index! Crawl-delay: 10