💾 Archived View for rawtext.club › ~sloum › geminilist › 000509.gmi captured on 2020-09-24 at 02:31:16. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
solderpunk solderpunk at SDF.ORG
Sun Mar 22 11:51:21 GMT 2020
- - - - - - - - - - - - - - - - - - - ``` On Sat, Mar 21, 2020 at 09:39:46PM -0400, Sean Conner wrote: > I don't mind the crawling, but I am concerned about the references to > robots.txt. In the web world, robots.txt lives at the top level and *only* > at the top level. I don't think there's been a official response from > solderpunk about robots.txt, but I would expect it to be very similar to how > it works on the web---the top level only. > > But a clarification would be nice (either way). In my opinion, it should > only live at the top level, but I can adapt to every "directory" as well. This is nicely timed, actually, as things like robots.txt are nowlooming larger on my personal radar than they have previously - withCAPCOM I am writing for the first time a program which automaticallymakes Gemini requests, and I'm very keen on making sure that it's a"good citizen". There hasn't been too much overt discussion of goodGemini citizenship yet, but now that non-human clients are becoming morecommon, there should be. Robots.txt is obviously part of that package. (It's *not* super relevant to feed aggregation, because nobody publishesa feed without the expectation that it is read entirely by bots, butother issues, especially rate limiting, rate) It's been many years since I read any robots.txt specs from the web. Iwill refresh my memory and start thinking about this, and askingquestions, in the hopes that we can finalise some stuff soon. Cheers,Solderpunk