💾 Archived View for rawtext.club › ~sloum › geminilist › 004939.gmi captured on 2024-02-05 at 11:16:51. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
Sean Conner sean at conman.org
Wed Jan 27 10:56:04 GMT 2021
- - - - - - - - - - - - - - - - - - -
It was thus said that the Great René Wagner once stated:
Hi,
simple question:
is the following robots.txt format valid in a form that the "disallow" is applied to all User-agents mentioned before?
---
User-agent: researcher
User-agent: indexer
User-agent: archiver
Disallow: about
---
That will work, but you need to add a leading '/' to the Disallow line:
Disallow: /about
That will match any request starting with '/about', like '/about','/aboutthis', '/about/that', etc.
or do i need to be more chatty?
---
User-agent: researcher
Disallow: about
User-agent: indexer
Disallow: about
User-agent: archiver
Disallow: about
---
That will work too (same thing about the Disallow: line though). You canread more about it at <http://www.robotstxt.org/>.
-spc