txt file is then parsed and can instruct the robot as to which web pages usually are not to generally be crawled. As a search engine crawler might preserve a cached copy of the file, it could every now and then crawl internet pages a webmaster does not wish to crawl. Internet pages typically prevented from currently being crawled include things lik