What Is A Search Engine Spider?

In search engine optimization, the most important animal is the spider. If you happen to read this blog on a regular basis, you may have heard me fondly refer to the search engine spiders quite often and how they crawl your site. If you aren’t quite clear on what a search engine spider is, you may find my consistent references quite creepy. I would like to take this time to weave the web of clarity on search engine spiders and (possibly) lower the level of creepiness that spiders bring you.

When a search engine ranks pages, they use a computer program and algorithm to browse the Internet in a methodical, automated manner. Some nicknames for this automated program are web crawler, ant, automated indexer, bot, and spider. The process of providing up-to-date data on web pages is called “web crawling”, so the nickname of “spider” seemed quite obvious.

Spiders are mainly used to create a copy of all the pages its visited for processing by the search engine, which indexes the page to provide fast search results. Spiders are also used for automating maintenance and gathering information from web sites, such as link checking, validating code and keywords. As spiders visit URLs, they identify all the hyperlinks for that page and add them to their list of sites to visit, as well as credit the original site for providing those links.

The information that the search engine spider gathers determines the page ranking of a website for its keywords. That is where search engine optimization comes in. Web pages that are optimized for their keywords in the elements and code make the spiders’ job much easier. Spiders like it when websites are optimized because it means they can work faster and more efficiently. We like spiders here at Volacci because they help us do optimize our clients’ better. We hope that you can shake your case of the heebie-jeebies and learn to love the spiders that are crawling your site as we speak.

Volacci is the leading Drupal SEO company and very passionate about your online success. By the end of your contract you will have at least as much additional business from your web site as you spend on our services... or we'll work for free until you do.

Volacci.® Your Profit. Our Passion.