Make Your XML Sitemap Work For You

14 November 2008
Sitemaps help people find what they are looking for, and in the case of Google's spider, Googlebot, the ability to orient themselves and manage their crawl activities.

Following on from one of our earlier posts on SEO (search engine optimisation), let's talk about XML sitemaps in more detail. Aside from registering your website with the major search engines, you should also have an up to date XML sitemap of your website. This file is essentially a list of all pages on your website.

Sitemaps help people find what they are looking for, and in the case of Google's spider, Googlebot, the ability to orient themselves and manage their crawl activities. Spiders such as Googlebot may complete the indexing of your website over many visits, and even after completing it's crawl of your website may return from time-to-time to check for changes. A sitemap gives the spider a rapid guide to the structure of your website and what has changed since it's last visit.

Googlebot will look at the depth of your sitemap amongst other factors to determine how to distribute your PageRank, the numerical weighting it assigns to the relative importance of your pages.

Step One: Create Your Sitemap

The easiest way to create your sitemap is to use an automated XML sitemap generator, such as the one at xml-sitemaps.com. This is a free service that will index up to 500 pages - you will need to pay them some money if you have more pages on your website...

Once it has generated your sitemap, you will need to download it to your computer (the file to download is the uncompressed XML file).

Step Two: Reference Your Sitemap In robots.txt

Once you have downloaded your sitemap, you need to reference it in your robots.txt file. This file tells spiders such as Googlebot what files and directories it is and isn't allowed to index. The robots.txt file needs to be in the root directory of your website. To reference your sitemap you need to add this line to it:

Sitemap: http://www.mywebsite.com/sitemap.xml

(obviously replace mywebsite.com with your real domain name!)

Step Three: Upload Your Files To Your Website

You now need to upload your sitemap.xml and robots.txt files to the root directory of your website.

Step Four: Tell The World About Your Sitemap!

Now that you have sorted your sitemap.xml and robots.txt files out, you need to submit your sitemap to the main search engines. In other words, you give the address to your sitemap (i.e. "http://www.mywebsite.com/sitemap.xml") to the search engines and request them to send out a 'ping' to it and 'listen' for a reply, kind of like the echo on a submarine's sonar search.

Here's how to submit your sitemap to the major search engines:

Google

* Create a Google account if you haven't already done so.
* Add Webmaster Tools to your account.
* Add your website - you will need to either insert a meta tag in your HTML code of your home page or upload a file to the root directory of your website to verify you own/are authorised to work on your website.
* Once you have done this, you can add your sitemap by clicking on the sitemap column link next to your website's name.

Yahoo!

Yahoo! follows a similar approach to Google, in that you need to create an account with them and then register your XML sitemap with them.

* Create a Yahoo! account if you haven't already done so.
* Register your website with Yahoo! Site Explorer and verify your website.

ask.com

Type in the following URL into your web browser's address bar:

http://submissions.ask.com/ping?sitemap=http://www.mywebsite.com/sitemap.xml

(obviously replace mywebsite.com with your real domain name!)

MSN/Live Search

Type in the following URL into your web browser's address bar:

http://api.moreover.com/ping?u=http://www.mywebsite.com/sitemap.xml

(obviously replace mywebsite.com with your real domain name!)

Note: if you are wondering who moreover.com is, then wonder no more! With MSN/Live Search, they have yet to implement a formal interface to submit XML sitemaps to. moreover.com are the official provider of RSS feeds to the MyMSN portal, so we are working on the theory that by moreover.com pinging your sitemap, it will tell MSN/Live Search about your XML sitemap somewhere along the line.