💾 Archived View for rawtext.club › ~sloum › geminilist › 004937.gmi captured on 2024-02-05 at 11:16:52. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2021-11-30)

-=-=-=-=-=-=-

<-- back to the mailing list

[tech] robots.txt format

Stephane Bortzmeyer stephane at sources.org

Wed Jan 27 10:27:41 GMT 2021

- - - - - - - - - - - - - - - - - - - 

On Wed, Jan 27, 2021 at 10:40:39AM +0100, René Wagner <rwagner at rw-net.de> wrote a message of 23 lines which said:

simple question:

Complicated answers:

is the following robots.txt format valid in a form that the
"disallow" is applied to all User-agents mentioned before?

1) There is no standard for robots.txt.

2) There is not yet an "official" adaptation to Gemini, justproposals.