💾 Archived View for rawtext.club › ~sloum › geminilist › 000508.gmi captured on 2020-10-31 at 01:37:53. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2020-09-24)

-=-=-=-=-=-=-

<-- back to the mailing list

Requests for robots.txt

Sean Conner sean at conman.org

Sun Mar 22 01:39:46 GMT 2020

- - - - - - - - - - - - - - - - - - - ```

  I'm going through my Gemini logs, and I'm finding this:

remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/filesystem.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/sample.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/userdir.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/msg.lua:1965/robots.txt" bytes=14 subject="" issuer=""remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/cgi.lua:1965/robots.txt" bytes=14 subject="" issuer=""

(I'm censoring the IP to protect the guilty here)

  I don't mind the crawling, but I am concerned about the references torobots.txt.  In the web world, robots.txt lives at the top level and *only*at the top level.  I don't think there's been a official response fromsolderpunk about robots.txt, but I would expect it to be very similar to howit works on the web---the top level only.

  But a clarification would be nice (either way).  In my opinion, it shouldonly live at the top level, but I can adapt to every "directory" as well.

  -spc