Robots.txt file is an old method to keep search engines from imporperly indexing your site. The method was developed in the mid 90's when technology was lesser to today. Currently, this strategy is no longer used frequently as search engine spiders and programs are desgined more sophisticated so that your site is protected against inaccurate indexing.