On 11/24/20 11:44 PM, John Cowan wrote: > 2) Though it does not respect robots.txt, it is happy to make your > content invisible to archive users by informal request (or, of course, > by a DCMA takedown notice). The Internet Archive does respect robots.txt, though they're not happy about it and have written on the subject a few times. I included a snippet in an earlier email with their user-agent.
---
Previous in thread (37 of 70): 🗣️ John Cowan (cowan (a) ccil.org)
Next in thread (39 of 70): 🗣️ Nick Thomas (gemini (a) ur.gs)