💾 Archived View for rawtext.club › ~sloum › geminilist › 000508.gmi captured on 2020-09-24 at 02:31:19. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Sean Conner sean at conman.org
Sun Mar 22 01:39:46 GMT 2020
- - - - - - - - - - - - - - - - - - - ``` I'm going through my Gemini logs, and I'm finding this: remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/filesystem.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/sample.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/userdir.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/msg.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/cgi.lua:1965/robots.txt" bytes=14 subject="" issuer="" (I'm censoring the IP to protect the guilty here) I don't mind the crawling, but I am concerned about the references torobots.txt. In the web world, robots.txt lives at the top level and *only*at the top level. I don't think there's been a official response fromsolderpunk about robots.txt, but I would expect it to be very similar to howit works on the web---the top level only. But a clarification would be nice (either way). In my opinion, it shouldonly live at the top level, but I can adapt to every "directory" as well. -spc