💾 Archived View for rawtext.club › ~sloum › geminilist › 004936.gmi captured on 2024-03-21 at 16:46:59. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2021-11-30)

-=-=-=-=-=-=-

<-- back to the mailing list

[tech] robots.txt format

René Wagner rwagner at rw-net.de

Wed Jan 27 09:40:39 GMT 2021

- - - - - - - - - - - - - - - - - - - 

Hi,

simple question: is the following robots.txt format valid in a form that the "disallow" is applied to all User-agents mentioned before?---User-agent: researcherUser-agent: indexerUser-agent: archiverDisallow: about---

or do i need to be more chatty?---User-agent: researcherDisallow: aboutUser-agent: indexerDisallow: aboutUser-agent: archiverDisallow: about---

kind regardsRené