Getting Google to crawl deeper
The first step to showing up in Google's search results is to have Google index your site. And you want Google to index every page of your site but this doesn't always happen. In fact, if you don't have a good enough page rank then Google probably won't index your whole site, especially if the site is large. One way you can increase your chances of being indexed is by keeping the search engine spiders away from your lower quality pages. You can block these spiders with proper robots.txt directives. This way the search engines only see the pages you want them to see and you're one step closer to ranking at the top of Google's search results.