Robots.txt

A Robots.txt file contains an instruction set that Search Engine Robots should follow when crawling your website content (but this is not always adhered to).  Robots should visit this file before anything else on the site.

Common items found in a Robots.txt file include disallowing certain bots from crawling your website content at all or excluding certain pages on your website from being crawled in particular.

Robots.txt is available for your COOLSite at www.YourDomain.com/robots.txt.