💾 Archived View for rawtext.club › ~sloum › geminilist › 000436.gmi captured on 2020-11-07 at 01:29:31. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

WWW indexing concerns (was: Gemini Universal Search)

Sean Conner sean at conman.org

Wed Feb 26 19:29:59 GMT 2020

- - - - - - - - - - - - - - - - - - - ```

It was thus said that the Great Andrew Kennedy once stated:
> 
> So the issue here is that the only way to opt out of being indexed is to
> contact each proxy maintainer and request that they make accommodations
> for you. That's fine with only 15 or so gemini servers, but not fair to
> proxy maintainers as gemini grows. It's also not enough to ask all proxies
> to use robots.txt, because there's nothing stopping someone from ignoring
> it either out of ignorance or in bad faith.

  There are other ways.  One way is to recognize a proxy server and blockany requests from it.  I think it would be easy to recognize one because ofall the requests from a single IP address (or block of IP addresses).  Theblocking can be at a firewall level, or the gemini server could recognicethe IP (or IP block) and close the connection or return an error.  That canbe done now.

  A second one is to extend robots.txt to indicate proxying preference, orsome other file, but then there are multiple requests (or maybenot---caching information could be included).  Heck, even a DNS record (likea TXT RR with the contents "v=Gemini; proxy=no" with the TTL of the DNSrecord being honored).  But that relies upon the good will of the proxy tohonor that data.

  Or your idea of just asking could work just as well.

  -spc