The Automatic Discovery Indexing Myth & Solution

It has been a year since I have started spending time learning SEO for some of my blogs and sites. I have read a lot posts and information available on the net. One myth I would like to bust once for all: a well designed site or blog with great content and quality inbound links is enough to help it take off in rankings. No, it is not! Automatic discovery by bots does not do a good job at all in many cases!

Some findings:

• Connection Between Users & Quality Content - This is a chicken and egg issue. Users like quality content, but need to be exposed to it. Quality content is determined by the number of users reading it. New websites or blogs with little content can be trapped in low rankings because of lack of exposure. It is a vicious circle, no matter the quality of the content.
• Boost For New Website - Some people mention that Google (or others) will give an exposure boost to pages of new websites/blogs to check their value against users. I could not confirm this claim, but even if it is true, there is a likely chance that your pages will not get enough ranking momentum from this, especially if there is a lot of ranking competition. During the 2010 Oil Spill in the Gulf of Mexico, some of the oil never reached the surface and stayed stuck in layers below the surface. It was not different than the oil which reached the surface.
• Social Buttons Do Not Differentiate Your Pages - They do not help that much, because everyone is using them. Hence, (nearly) everyone is on the same level playing field. They only differentiate against websites not using them. So use them, but only to avoid the penalty of not using them.
• Conflicting Needs When Creating Links - Search engines recommend building links which make sense for the user, not for ranking. On the other side, this recommendation does not help creating a network of links covering all your pages. You could create menus, but this can be tedious if you are not using a content management system or if you are dealing with a blog.
• Google Does Not Index Pages It Cannot Crawl - If the set of external and internal links does not cover all your pages transitively, some pages will never be indexed. Moreover, pages which are only reachable in a transitive way may be indexed long after they have been made available on the net.

To make it short: unreachable pages are never indexed, hardly reachable pages are indexed very slowly, lack of traffic means less bots crawling your site, which slows down the indexing process and pages can remain stuck below the ranking surface for a long time because of a lack of exposure.

I did notice that proper usage of keywords, including in the URLs, does help with exposure, but only once the pages have been crawled and indexed. It is one part of the solution. But, the real silver bullet to accelerate crawling (which is necessary for indexing) and exposure for better ranking is the use of complete sitemaps with all the necessary URLs to all your content. You can submit them in Google Webmaster Tools. For Blogger blogs, you can use atom sitemaps.

This makes sure Google will have a fast access to all your content for crawling, meaning fast indexation too. Google Webmaster will tell you how much of these sitemaps has been processed and report any issue. More indexation earlier, means more exposure to traffic earlier. If you have great content, users will see it faster and are more likely to use social buttons or to create external links to your content. They will also come back more often. I have observed that quality content rises to the surface faster.

If, after setting sitemaps, your site still does not take off, it is probably because no one is interested in your content. At least, you will know it is not because of an indexing or ranking issue (and you will know it pretty quickly).

TL, DR: Hoping bots will automagically discover your best pages and necessarily give them the indexing, ranking and exposure they deserve is nothing more than wishful thinking, especially on the long run. Do create and submit complete sitemaps to provoke fast distribution of fairness.

More tips & tricks about sitemap creation here.