No one is perfect. And neither is Drupal’s default robots.txt file. In fact, there are several problems with the file. If you test out your default robots.txt file line by line using Google Webmaster Tools’ robots.txt testing utility, you will find that a lot of paths which look like they are being blocked will actually be crawled.
In a scene that could be described more accurately with hand gestures than with words, Search Engine Optimization officially eliminated the robots from the White House last week. It was one promise fulfilled, and a move towards change. Change that doesn’t just buy you double the beef, but a new beginning for our President’s new home web page. Within the first day of the Obama administration, a redesign of the White House’s website removed the robots.txt coding and drastically increased its search engine optimization. It is commonly believed that much of the website for the White House (whitehouse.gov) under the Bush’s administration was not crawl-able by search engine spiders due to its coding.