💾 Archived View for rawtext.club › ~sloum › geminilist › 000510.gmi captured on 2020-11-07 at 01:32:43. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2020-09-24)

-=-=-=-=-=-=-

<-- back to the mailing list

Requests for robots.txt

Natalie Pendragon natpen at natpen.net

Sun Mar 22 11:59:22 GMT 2020

- - - - - - - - - - - - - - - - - - - ```

FWIW I'm 99% sure those are requests from GUS, and I agree that itshould be top level only. That was a regression in GUS' crawling code,which I've now fixed!

I'm still very happy to accommodate more official guidance on howrobots.txt should work, but in the meantime (and in the absence of anymore regressions, eep!) I plan to check top-level-only robots.txt.

So sorry about the :bug:!