How to Fix the Problems with Drupal’s Default Robots.txt File
Find out if your robots.txt file is following current best practices & that it's doing what you think it's doing. You may just find that many paths you thought were being blocked are actually getting crawled by the search engines. We can show you how to resolve those issues.