💾 Archived View for station.martinrue.com › freezr › e7837e28023940f7afd24beb3a1d2f5e captured on 2023-11-14 at 10:22:49. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-09-08)
-=-=-=-=-=-=-
Dear search engine makers how can I tell your services to do not crawl some specific folders?
I use a folder where I upload the drafts and I don't want that discoverable.
Can you help me with this? 🙏
Thanks! 👍
1 year ago
Its pretty easy. As other's have said, putting a "robots.txt" in the root of our capsule is what you need. It contains lines that tell crawlers what URLs to ignore. Solderpunk defined that Gemini uses a really primative version of Robots.txt. Here is my:
gemini://gemi.dev/robots.txt
Any URLs that start with the text listed on a line will be ignored. So if you want to have a crawler ignore anything inside of "gemini://example.com/drafts/" put this in:
user-agent:*
Disallow: /drafts/ · 1 year ago
@moddedbear thanks! 👍🍺 · 1 year ago
Robots.txt is way simpler than most pages in search results make it out to be I'm just noticing. · 1 year ago
You could start it off with "User-agent: *" to target all crawlers and then if you wanted to disallow a directory called drafts the next line should be "Disallow: /drafts/". · 1 year ago
@moddedbear I did not understand how to use it...
My level of abstraction doesn't decode information laid off that way... 😩😩😩 · 1 year ago
I don't think you typically have to worry as long you're not linking to your drafts anywhere, but you should look into putting a robots.txt in your capsule root. I'll let you look up the specifics but it's a pretty simple text file that tells crawlers where they should or shouldn't go. · 1 year ago