Maximize Search Engine Crawling
Possible titles/subheads: Make it easy for search engines OR Help search engines help you
Search engines will not spend an infinite amount of time on your site and they do not crawl most sites on a daily basis.
Search engines crawling your site is a fundamental and important piece of SEO. In order for your website and its pages to show up in search results, they must first be crawled by the search engine and subsequently get indexed. The faster and more thoroughly the search engines crawl your site, the faster your pages can be indexed and begin to rank well. Also, wouldn't it be nice for you to make some sort of improvement to your website and have the search engines notice faster, and hopefully rank you higher soon?
A SEOBook.com post suggests that a website's last cache date (when Google last crawled the site) is a good indicator of Google's algorithmic opinion of that website, and better than PageRank.
Want to know when Google, Yahoo and Bing last crawl your site and how many of your site's pages they currently have in their index? Look at...
Have a quality site that search engines value
Contrary to previous belief, Matt Cutts of Google said in a March 2010 interview, "There is also not a hard limit on our crawl. The best way to think about it is that the number of pages that we crawl is roughly proportional to your PageRank."
This means that having a site with more valuable backlinks and PageRank can increase the amount of time that Google will spend crawling your site, and how often they check back.
Help search engines find the pages that are important to you
- Make sure your website has logical navigation and a good linking structure that means a person or bot can get to all your important pages in less than 3 clicks from your home page.
- Have a thorough and up-to-date HTML and/or XML sitemap that:
- contains all your pages or the most important pages on our website.
- search engines can find. You can link to it on your home page or tell search engines where it lives in your robots.txt file, for example.
Don't send search engines to place you don't care about
- NoFollow links to pages you don't care about, like your privacy policy or sign in screen. You can do this in the link by adding rel=nofollow like this: <a href="http://www.example.com/" rel="nofollow">anchor text</a>
- NOINDEX pages you don't want search engines to include in search results. You can do this at the page level, or ask search engine bots not to index certain types of pages via your robots.txt file.
