Find out if your robots.txt file is following current best practices & that it's doing what you think it's doing. You may just find that many paths you thought were being blocked are actually getting crawled by the search engines. We can show you how to resolve those issues.
In a scene that could be described more accurately with hand gestures than with words, Search Engine Optimization officially eliminated the robots from the White House last week. It was one promise fulfilled, and a move towards change. Change that doesn’t just buy you double the beef, but a new beginning for our President’s new home web page. Within the first day of the Obama administration, a redesign of the White House’s website removed the robots.txt coding and drastically increased its search engine optimization. It is commonly believed that much of the website for the White House (whitehouse.gov) under the Bush’s administration was not crawl-able by search engine spiders due to its coding.