Posted to Ben Finklea's blog on August 11th, 2008

Obstacles to search engine crawling

Before your site can be indexed it has to be successfully crawled by the search engines. The search engines crawl through your website looking for a valid URL. If the search engine finds the URL it can then index your website. That is why it is important for you to make sure that the search engines can find your URL. You must be careful with your use of JavaScript links, sometimes the engines don't recognize these links. Some websites will also block search engine robots (sometimes purposefully, sometimes not) and this will definitely keep them from successfully finding your URL.
To make your site indexable you must allow robots and provide easy-to-read links. To get your website in the search engine results it is vital that it be properly indexed and the best way to ensure that it is indexed is by making it easy to crawl.