Posted to Ben Finklea's blog on September 18th, 2008

Getting Google to crawl deeper

The first step to showing up in Google's search results is to have Google index your site. And you want Google to index every page of your site but this doesn't always happen. In fact, if you don't have a good enough page rank then Google probably won't index your whole site, especially if the site is large. One way you can increase your chances of being indexed is by keeping the search engine spiders away from your lower quality pages. You can block these spiders with proper robots.txt directives. This way the search engines only see the pages you want them to see and you're one step closer to ranking at the top of Google's search results.