Sean Conner sean at conman.org
Mon Feb 22 01:32:27 GMT 2021
- - - - - - - - - - - - - - - - - - -
It was thus said that the Great Emma Humphries once stated:
What's the use case behind blocking proxies?
Some people don't want web crawlers to crawl their Gemini space. Webcrawlers can be ... stupidly bad [1]. For me, it's for web proxies that*DON'T* have a robots.txt preventing web crawlers through the proxy.
-spc
[1] http://boston.conman.org/2019/07/09-12