Week 9: 10 Things to do Before the Spiders Come

Tip #9: The robots.txt

This file allows you to filter out the bots that crawl your website, plus you can restrict certain URLs that do not need to be indexed. This is a tremendous resource if used properly.

Comments

Post new comment

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

The Volacci SEO Podcast

Ben Finklea talks with Stephen and Thomas from Tree House Agency and Robert Douglass of Acquia at Drupalcon 2009 in Washington DC.

The Volacci SEO Blog

When you are ready to launch a search marketing campaign, there are two distinct, results-driven...

Partner With Volacci

Being a partner with Volacci has many advantages

By participating in our referral program, you will financially benefit from any referrals you send our way.