Learn/Get-Your-Website-Indexed

Learn

Being indexed is the key to showing up in search engines

By Kristina Weis
email or post a comment for me

It's something that every website must go through. Going from a new, obscure website that can only be found by someone who knows its URL to one whose pages are indexed and can be found by people searching for relevant keywords in search engines. After creating a solid website, being found by search engines is the first step...

The sooner a website can get indexed, the sooner it can begin ranking well in search engines (SEO) and receiving more traffic and business.

Is your website and its pages indexed?

Try searching in Google for "site:yourwebsite.com" and see what turns up. This isn't 100% accurate, but it can give you a rough idea of what you're working with. This will tell you which pages are in the index, and usually if it's a little off it will show less pages than are actually in the index.
If you've submitted an XML sitemap via Google Webmaster Tools you can find the most accurate number of pages in your sitemap that Google has indexed. This will tell you the number of pages in Google's index out of the total number of pages in your sitemap, but it won't tell you which pages are indexed.
Use this tool...

Ways to Help Your Website and its Pages Get Indexed Sooner

Have a website with unique, valuable content that you would find valuable enough to serve in search results if you were a search engine. I hope this is obvious. A website without quality content is unlikely to get indexed no matter how hard someone tries the other tactics below.
Get inbound links to your site. The primary way that search engines find new sites and web pages to add to its index and search results is by following links from other websites. Links without a NoFollow attribute (often called DoFollow links) are best, but NoFollow links may still be helpful (particularly ones from popular places like Twitter).
- Ask a friend or related website to link to your site from theirs or in a blog post
- Get a DoFollow link from AboutUs.org
- Create additional web presences for your website. There's a list of directories and similar sites in my Reputation Management article.
If your site is brand new and doesn't have any pages in search engines yet, submitting it to search engines can help. Here is Google's submission form. Don't, however, waste your time and/or money with so-called search engine submission services.
Creating an HTML sitemap can help search engines crawlers find and index more pages on your site, once they've found your website and "sitemap" link.
An XML sitemap can also help search engines index more of your web pages, and in addition it allows you to tell search engines which pages are most important to you.
Setting a custom crawl rate in Google Webmaster Tools asks Google to crawl pages more quickly, which can be helpful for very large websites. If you do this, make sure that it doesn't hinder the performance, speed or usability of your site.

Double check:

If none of your website's pages are showing up in search engines, make sure you aren't accidentally telling them not to.

Look at the page source (in most web browsers, clicking ctrl and the letter U will do this) for one of your web pages and make sure that there isn't something like in there. If a "NOINDEX" tag is there, you will need to remove it because that tells search engines to go away and not put that page in its index.
Look at yourwebsite.com/robots.txt and make sure you aren't telling search engine spiders to stay out and not index a certain type of page with something like this: User-Agent: * Disallow: /Special/ (This as an example, tells all search engines not to index pages on AboutUs.org that have a URL like AboutUs.org/Special/something.)
Crawl errors can slow down the rate at which search engines are able to spider and index a website. You can find these in Google Webmaster Tools, along with more details about the specific errors they are experiencing. For very large websites with lots of crawl errors, search engines may decide to stop and not index the entire site.
There's no good way to know for sure if Google or another search engine has penalized or banned your site, but... If an established website -- one that's been around for a while and has some decent inbound links -- has noticed a drop in traffic from that search engine and/or doesn't have pages indexed in search engines (like a Google search for "site:example.com" turns up nothing), a search engine penalty may be possible. Check Google's quality guidelines for a list of things to avoid and that they may penalize sites for.

See more articles like this one.