On 24.11.2020, marc wrote:
> I suppose I am chipping it a bit too late here, but I think
> the robots.txt thing was always a rather ugly mechanism - a
> bit of an afterthought.
+1 that the robots.txt solution feels a lot like a hack.
> So the way I remember it, robots.txt was a quick hack
> to prevent spiders getting trapped in a maze of
> cgi generated data, and so hammering the server.
> It wasn't designed to solve matters of privacy
> and redistribution.
There is a more modern alternative to robots.txt which is the X-Robots-Tag
HTTP header and sounds like what you are trying to do here.
That said, there are probably people who will not want special headers to be
added [1], altough I personally think that something like you suggest would not
be that "exploitable". Especially because it is just part of the documents text.
[1] See the first sentence of ?2.4 of the Gemini FAQ
gemini://gemini.circumlunar.space/docs/faq.gmi
https://gemini.circumlunar.space/docs/faq.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201124/ebbb
436e/attachment.sig>
---
Previous in thread (20 of 70): 🗣️ marc (marcx2 (a) welz.org.za)
Next in thread (22 of 70): 🗣️ Philip Linde (linde.philip (a) gmail.com)