💾 Archived View for rawtext.club › ~sloum › geminilist › 007339.gmi captured on 2023-11-04 at 12:36:47. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
indieterminacy at libre.brussels indieterminacy at libre.brussels
Thu Oct 21 16:48:15 BST 2021
- - - - - - - - - - - - - - - - - - -
Stephane,
Perhaps we can employ a stricter standard concerning poorly functioning bots?
For example, a federated 'naughty-step':* If a server is being bothered by cretinous behaviour message a naughty-step site* From time to time (participating) servers get a list of ips to (temporarily!) block* The bots receive a (new) response code, pointing out that it is temporarily on a naugty-step* The response code recommends a uri to better bot practices
Jonathan
Stephane Bortzmeyer <stephane at sources.org> writes:
On Wed, Oct 20, 2021 at 09:19:49PM -0400,
Sean Conner <sean at conman.org> wrote
a message of 33 lines which said:
* disrespecting robots.txt, the protocol spec and common sense
Yeah, tell me about it. Here's the robots.txt I had (have):
User-agent: *
Disallow: /test/redirhell
Speaking of specifications, remember that robots.txt is not
standardized anywhere, even when you limit yourself to the Web (and it
is worse for Gemini, which has no User-Agent: field). There are
several documents floating on the net and the examination of the
robots.txt of Gemini capsules indicate a lot of variety.
(Your robots.txt is very simple and therefore fits into the common
subset of all robots.txt documentation/software. But it is not the
case for all robots.txt.)