User-agent: * Crawl-delay: 30 # Non-Standard Request-rate: 1/5 # (proposed) Extended Standard for Robot Exclusion; one page every 5 seconds