User-agent: * Noindex: /user/ Disallow: /user/ Allow: / Crawl-delay: 40 User-agent: Slurp Disallow: /user/ Allow: / Crawl-delay: 0.6