Don't Block Search Engine Crawlers
By [[User:|]] on
Make sure your important pages can rank
A great page on your website will never rank well in search results - and be discovered by more people - if the search engines don't know it's there in the first place.
First, some background: Search engines use computer programs called "crawlers" (or “spiders” or “robots”) to discover web pages and add them to their indexes. Google calls its crawler “Googlebot,” and lots of SEO (search engine optimization) professionals call it that, too.