Great news from yesterday at SES. Danny has a great roundup describing that Search Engines Unite On Sitemaps Autodiscovery at Search Engine Land and I have some more details with my coverage of the Sitemaps & URL Submission session from yesterday.
In short, all four major search engines, Google, Yahoo, Microsoft's Live.com and Ask.com will all support an autodiscovery method for Sitemaps. Sitemaps is an XML protocol that enables you to freely submit a listing of URLs with more meta-data to the search engines, so that the engines can be assisted in their crawl process. It is like a form of paid inclusion without paying.
Sitemaps was first introduced in November 2006 but back then you had to manually go to Google Webmaster Central or Yahoo Site Explorer and inform them about your sitemap. Now, all you need to do is put a little marker in your robots.txt file, telling the search engines the location of your sitemap and presto, the search engines will find it on their own.
Microsoft and Ask.com both promised to support it, but I believe are currently not supporting it yet.
More details on these bot sitemaps (not human sitemaps) at sitemaps.org.
Forum discussion at WebmasterWorld & Cre8asite Forums.
