Stephane Bortzmeyer stephane at sources.org
Thu Oct 21 09:51:36 BST 2021
- - - - - - - - - - - - - - - - - - -
On Wed, Oct 20, 2021 at 09:19:49PM -0400, Sean Conner <sean at conman.org> wrote a message of 33 lines which said:
* disrespecting robots.txt, the protocol spec and common sense
Yeah, tell me about it. Here's the robots.txt I had (have):
User-agent: *
Disallow: /test/redirhell
Speaking of specifications, remember that robots.txt is notstandardized anywhere, even when you limit yourself to the Web (and itis worse for Gemini, which has no User-Agent: field). There areseveral documents floating on the net and the examination of therobots.txt of Gemini capsules indicate a lot of variety.
(Your robots.txt is very simple and therefore fits into the commonsubset of all robots.txt documentation/software. But it is not thecase for all robots.txt.)