💾 Archived View for rawtext.club › ~sloum › geminilist › 005422.gmi captured on 2023-11-14 at 10:06:42. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2021-11-30)

-=-=-=-=-=-=-

<-- back to the mailing list

[SPEC] Encouraging HTTP Proxies to support Gemini hosts self-blacklisting

Sean Conner sean at conman.org

Mon Feb 22 01:32:27 GMT 2021

- - - - - - - - - - - - - - - - - - - 

It was thus said that the Great Emma Humphries once stated:

What's the use case behind blocking proxies?

Some people don't want web crawlers to crawl their Gemini space. Webcrawlers can be ... stupidly bad [1]. For me, it's for web proxies that*DON'T* have a robots.txt preventing web crawlers through the proxy.

-spc

[1] http://boston.conman.org/2019/07/09-12