💾 Archived View for diesenbacher.net › blog › entries › gemini-robots-txt.gmi captured on 2023-06-14 at 13:56:09. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-01-29)
-=-=-=-=-=-=-
Looking into my gemini capsule log revealed, that - probably - the geminispace.info bot looked for a file robots.txt as it crawled my capsule. Oh, well known robots.txt in the Miasma-world of the big Internet :), I was not aware, that also gemini-bots expect such a file.
A quick search found: [robots.txt for Gemini].
following my currently completey useless robots.txt, as I want the bot to visit anything, actually the "Allow:" isn't even defined, but shouldn't hurt:
,----
| # robots.txt for my capsule
|
| Allow: *
|
| # ;)
`----
But for the future, as I plan some CGI-like features and "gemini-app", this makes absolutely sense, to prevent the bot to access sensible links.
Quite interesting remark in the link above about robots.txt: there are - for sure - no user-agent properties in a gemini request.