Stephane Bortzmeyer stephane at sources.org
Wed Jan 27 11:12:54 GMT 2021
- - - - - - - - - - - - - - - - - - -
On Wed, Jan 27, 2021 at 05:56:04AM -0500, Sean Conner <sean at conman.org> wrote a message of 33 lines which said:
That will work too (same thing about the Disallow: line though). You can
read more about it at <http://www.robotstxt.org/>.
But do note that many Gemini capsules do not follow this specificationbut one of the others (typically more complicated).