💾 Archived View for rawtext.club › ~sloum › geminilist › 004940.gmi captured on 2023-09-08 at 17:32:58. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2021-11-30)

-=-=-=-=-=-=-

<-- back to the mailing list

[tech] robots.txt format

Stephane Bortzmeyer stephane at sources.org

Wed Jan 27 11:12:54 GMT 2021

- - - - - - - - - - - - - - - - - - - 

On Wed, Jan 27, 2021 at 05:56:04AM -0500, Sean Conner <sean at conman.org> wrote a message of 33 lines which said:

That will work too (same thing about the Disallow: line though). You can
read more about it at <http://www.robotstxt.org/>.

But do note that many Gemini capsules do not follow this specificationbut one of the others (typically more complicated).