Website spidering ability
A website cannot rank well in search engines unless it has been properly crawled and indexed by search engines. A proper form means that the search engine crawler can reach all the pages of the website. In other words, the spiderability of the website can be termed as a good and effective navigation structure of the website.
There are two important factors that need to be considered to improve the spiderability or indexing of a website:
1. Navigation has real hyperlinks
2. Distance of the deep link from the home page
navigation links
All websites have a navigation structure that allows users to navigate through the website. The question is, is the navigation effective and linked correctly and are these links a standard hyperlink? Standard hyperlinks are text or image links with all the SEO ingredients, ALT text, link title tag, etc.
Non-standard or poor navigation links are those that are embedded in links generated by Flash or JavaScript. Although, there is nothing wrong with such links, but not all search engine crawlers can pick up hyperlinks generated by JavaScript or links within flash.
Search engine crawlers that can select these links are very few. Google is one of them, but it is not clear if Google places any value on such links. The point here is that the Google Page Rank factor is mainly based on the links pointing to a page and not a single link can be ignored.
The navigation structure must be carefully created, since all the pages of a website must be linked to each other. It does not mean that each page should point to every page on the website, but it is recommended that each page should point to those pages that are most important.
deep link distance
The home page of a website is the first page a user will visit. Not only a user, but also the search engine spiders will crawl you first and choose the links that are placed on the home page.
Ideally; a user will click no more than twice to check out her services or products. The more clicks it takes to reach a page, the lower the chance of exposure. Similarly, deep links are less likely to be indexed by a search engine crawler if they are too far from the home page.
Use of the site map
It is not recommended to clutter the home page with many links and to avoid this, sitemaps are used. Sitemaps have become an important part of a website. By placing them within a click away from the home page, search engine crawlers can pick up a sitemap, and since a sitemap contains all the links on a website, the crawler can index the website complete with ease.
While using the sitemap to crawl your website can be a good idea, but the most effective way is to properly structure your website navigation.
