> Why are we defining new standards and filenames? bots.txt, .well-known, > etc. Just want to point out that the .well-known path for machine-readable data about an origin is a proposed IETF standard that has relatively widespread use today. It is filed under RFC8615, and is definitely not a standard that was invented in this thread. The first paragraph of the introduction even references the robots file. While I don't necessarily agree with the naming of bots.txt, I see no problem with putting these files under a .well-known directory. > We don't need this. Thanks for making this mailing list a lot more efficient and talking about what the Gemini community needs in a 4-word sentence. Even if the original path of /robots.txt is kept, I think it makes sense to clarify an algorithm in non-ambiguous steps in order to get rid of the disagreements in edge-cases. > let's just pick a standard that works and make it offical. The point is that the standard works for simple cases, but leaves a lot to be desired when it comes to clarifying more complex cases. This results in a lot of robots.txt implementations disagreeing about what is allowed and not allowed. Additionally by crawling the Web, you can see that people tend to extend robots.txt in non-standard ways and this only gets incorporated into Google's crawlers if the website is important enough. > I am no big fan of Google, but they are the kings of crawling and it > makes sense to go with them here. The kings of crawling deemed HTTP and AMP the most suitable protocol and markup format for content to be crawled, why don't we stop inventing standards like Gemini and Gemtext and go with them here. > The spec makes many example references to HTTP, but note that it is > fully protocol-agnostic, so it works fine for Gemini. Gemtext spec makes references to Gemini, but it is fully protocol-agnostic, so it works fine with HTTP. Similarly, Gemini makes many references to Gemtext, but it is content-type agnostic so it works fine with HTML. But we thought we could be better off shedding historical baggage and reinvented not one, but two main concepts of the traditional Web. -- Leo
---
Previous in thread (34 of 41): 🗣️ Petite Abeille (petite.abeille (a) gmail.com)
Next in thread (36 of 41): 🗣️ Stephane Bortzmeyer (stephane (a) sources.org)