crawling, proxies, and robots.txt

Hello,

Both of these projects and ideas sound great! Maybe it's obvious, but just in
case: Make sure you respect the user agent of * as well, which signifies any
robot.


makeworld

---

Previous in thread (3 of 4): 🗣️ Michael Lazar (lazar.michael22 (a) gmail.com)

View entire thread.