Natalie Pendragon natpen at natpen.net
Sun Mar 22 11:59:22 GMT 2020
- - - - - - - - - - - - - - - - - - - ``` FWIW I'm 99% sure those are requests from GUS, and I agree that itshould be top level only. That was a regression in GUS' crawling code,which I've now fixed! I'm still very happy to accommodate more official guidance on howrobots.txt should work, but in the meantime (and in the absence of anymore regressions, eep!) I plan to check top-level-only robots.txt. So sorry about the :bug:!