Well behaved spiders, like the ones Google and Yahoo use, will look at your robots file to decide which pages to index. This is an excellent way to steer the search engines toward the pages you would like to see in the search engine results (like content pages) and steer them away from pages that aren't as important (like administrative pages).

Be sure that you validate your robots file (robots.txt) by entering your full URL. Once your file has been validated you should be able to easily share your optimized pages with the search engines you are targeting.

Tip #10: Caches

If you do not want Google to cache a website, simply include this line of code between your head tags:

META NAME="ROBOTS" CONTENT="NOARCHIVE"

Thanks for reading every week. To review past weeks, simply visit the category: 10 Things to Do Before the Spiders Come. With these simple tips, your site can be properly indexed in no time!

Tip #9: The robots.txt

This file allows you to filter out the bots that crawl your website, plus you can restrict certain URLs that do not need to be indexed. This is a tremendous resource if used properly.

Tip #8: Keep your content fresh.

Google likes to index pages more frequently when they are updated on a regular basis. It’s also a great way to keep visitors returning when you offer them fresh new content. Blogs, articles, and new products/special offers are all a great way to keep Google - and customers - coming back for more.

Tip #7: Question- Do you have dynamically generated pages?

Google claims they will be limiting the number of dynamic webpages they will index. It may help if you include some static content in your pages.

Tip #6: Check your frames.

If you use frames on your website, it may not be fully indexed. Google recommends that you check out this article by Danny Sullivan called Search Engines and Frames.