User-agent: * # allowing do/index! Crawl-delay: 10