Posted to Ben Finklea's blog on January 21st, 2010

How to Remove a Web Page From Search

I know what you are thinking, and I don’t fail to see the irony in this post. After years of helping web sites and pages show up and rank well in all the major search engines, I will now tell you how to remove a web page from search results. There are actually a few good reasons for doing this, i.e. duplicate page, page with sensitive data, as well as a few different degrees to which you can remove your page from querying results. Just make sure you are doing it for “whitehat” reasons. Here are the several different ways to remove a web page or specific URL from search.

Password Protect a Page

Do you have a web page that has sensitive information or data that you want to keep from showing up in search results? Require a user ID and password for the web pages you want protected and Google’s spiders will be unable to access them.

Add the Noindex Tag

When the search engines see the noindex tag on a page, they won’t index the page. Best practices recommend that you put the noindex tag in the header area of the code on the page, but you can add it to another part of the page and it should still work.

Add to Robots.txt File

Your site should have a robots.txt file, Volacci’s robots.txt file is here. Make sure to add the “disallow” crawling/indexing to your file.

Remove All Inbound Links

If there are still links pointing at a web page, then the page will most likely stay in the search engine index. Get rid of all the inbound links that point to the page you want to remove, both internal links from within your site and external links from other sites. You may even need to ask the web masters of your external links to stop linking to said page.

Completely Remove the Page From Your Site

If you want your page to be completely removed from existence, the first step you need to take is remove or delete the page from your web server. Then, use the Google Removal Tool to delete the page from Google’s search index. You will need either FTP access to your site or the content management system to do this.

Using the Google Removal Tool, you will first need to verify your website in Google Webmaster Tools. This proves that you are the actual owner, and will need a Google Account to do so. Log into Google Webmaster Tools and “add your site”, then click on the “verify your site” link they provide. You will be asked to either add a file or meta tag to your website. Once your site is officially verified in Google Master Tools, you will need to do the following:

1. Click on “Site Configuration” then “Crawler Access” and then “Remove URL”

2. After clicking “Remove URL”, submit a new request from the several options provided: “Individual URLS”, “Directory or Subdirectory”, “Entire Site”, or “Cached Copy”

What you decide to remove depends on whether you want to remove one URL or an entire section of your site. There are several different options, and degrees, to which you can remove a web page from the attention of search results. In most cases, you may find that more than one of these options may be necessary to completely remove a page.