💾 Archived View for rawtext.club › ~sloum › geminilist › 007330.gmi captured on 2023-11-14 at 08:41:06. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
Stephane Bortzmeyer stephane at sources.org
Thu Oct 21 09:51:36 BST 2021
- - - - - - - - - - - - - - - - - - -
On Wed, Oct 20, 2021 at 09:19:49PM -0400, Sean Conner <sean at conman.org> wrote a message of 33 lines which said:
* disrespecting robots.txt, the protocol spec and common sense
Yeah, tell me about it. Here's the robots.txt I had (have):
User-agent: *
Disallow: /test/redirhell
Speaking of specifications, remember that robots.txt is notstandardized anywhere, even when you limit yourself to the Web (and itis worse for Gemini, which has no User-Agent: field). There areseveral documents floating on the net and the examination of therobots.txt of Gemini capsules indicate a lot of variety.
(Your robots.txt is very simple and therefore fits into the commonsubset of all robots.txt documentation/software. But it is not thecase for all robots.txt.)