User-agent: * Disallow: /article/in* Sitemap: http://www.newscientist.com/sitemap_index.xml.gz User-Agent: Zibber Disallow: /auth/ Disallow: /commenting/ Disallow: /contact/ Disallow: /info/ Disallow: /search* Disallow: /subs/ Disallow /search*offset=* Allow /search*offset=10 Allow /search*offset=20