💾 Archived View for bbs.geminispace.org › s › Cosmos › 665 captured on 2023-11-04 at 13:41:49. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-09-08)
-=-=-=-=-=-=-
Currently, there's the ability to limit displayed posts to recent ones, which is great.
On this page, there's a dynamic link to only display newer ones than the ones displayed on the recent posts page:
Unfortunately, this link is crawled by crawlers and ends up on search engine result pages. This is not ideal, because if I search for an uncommon keyword, where I can estimate the number of expected results, e.g. my last name, then I still get ~800 results, and the vast majority of them are links to Cosmos' "Show newer posts than this":
[deleted Geminispace.info search link for nordmeyer, which resulted in indexing the SERP because of a typo in geminispace.info/robots.txt]
Blocking `/~Cosmos/since` in robots.txt seems like a good idea.
Any thoughts on this?
2023-05-21 · 5 months ago
Agreed, I'll add that to robots.txt. That should alleviate the issue at least.