Posted to Ben Finklea's blog on July 9th, 2007
Week 9: 10 Things to do Before the Spiders Come
Tip #9: The robots.txt
This file allows you to filter out the bots that crawl your website, plus you can restrict certain URLs that do not need to be indexed. This is a tremendous resource if used properly.