Crawlers on Gemini and best practices

On Thu, Dec 10, 2020 at 03:20:18PM +0100,
 Petite Abeille <petite.abeille at gmail.com> wrote 
 a message of 8 lines which said:

> > On Dec 10, 2020, at 15:15, Stephane Bortzmeyer <stephane at sources.org> wrote:
> > 
> > Because with your algorithm, Google would be disallowed
> 
> An empty Disallow means allow all, no?

Yes, but the next Disallow add a restriction to everything.

---

Previous in thread (13 of 41): 🗣️ Philip Linde (linde.philip (a) gmail.com)

Next in thread (15 of 41): 🗣️ Stephane Bortzmeyer (stephane (a) sources.org)

View entire thread.