Writing a crawler is the most annoying thing ever, lol
3 years ago · 👍 bronzie_beat
I'm not really struggling with anything, it's just annoying to deal with some of the things. I have just added a crawl-delay feature, as well as making the crawler obey slowdown responses from the server, and doing that in a crawler that uses concurrency was a bit annoying, but I got it figured out. :D · 3 years ago
Anything in particular you're struggling with? · 3 years ago