💾 Archived View for rawtext.club › ~sloum › geminilist › 000517.gmi captured on 2020-10-31 at 01:38:14. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

robots.txt for Gemini

solderpunk solderpunk at SDF.ORG

Tue Mar 24 20:00:52 GMT 2020

- - - - - - - - - - - - - - - - - - - ```

On Tue, Mar 24, 2020 at 08:47:53AM -0400, Jason McBrayer wrote: 
> Another type I can think of is "archive", for things that rehost
> existing gemini content elsewhere on gemini. Besides being another use
> case, this category also has the implication that it may make deleted
> content available (a la the Wayback Machine).

Yes, certainly!  This is actually quite an important use case.  Manyfolks are unhappy that archive.org no longer respects robots.txt, whileon the other hand the archive.org folks argue that people were writingrobots.txt rules based on how they wanted search engine robots ot actrather than considering archive bots.  Making it easy for Gemini serveradmins to explicitly set different policies for the two kinds of bots(if they want to!) seems a substantial improvement.

Cheers,Solderpunk