💾 Archived View for station.martinrue.com › krixano › 5ad9d04a153342a49170456ca61a9f05 captured on 2023-03-20 at 20:29:42. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2023-01-29)

➡️ Next capture (2023-07-22)

-=-=-=-=-=-=-

👽 krixano

Writing a crawler is the most annoying thing ever, lol

1 year ago · 👍 bronzie_beat

Actions

👋 Join Station

2 Replies

👽 krixano

I'm not really struggling with anything, it's just annoying to deal with some of the things. I have just added a crawl-delay feature, as well as making the crawler obey slowdown responses from the server, and doing that in a crawler that uses concurrency was a bit annoying, but I got it figured out. :D · 1 year ago

👽 marginalia

Anything in particular you're struggling with? · 1 year ago