Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009 - 8:38 am 1 by
Filed Under Bing Search

Here is one more oddity to add to Microsoft Bing's web crawler, MSNBot. Why on earth are people reporting that MSNBot is crawling the same page twice, once for the compressed version and then once again for the uncompressed version? Technically, it should probably only crawl once and it should opt for the compressed, gzip version - don't you think?

We have two threads complaining about this, one oldish one at WebmasterWorld and another at Bing Forums. Let me quote the Bing thread:

I've notice that bing is crawling each page of my website twice, first making an HTTP 1.1 request and getting a compressed response then immediately issuing an HTTP 1.0 request to receive the same page without gzip compression The following lines from my log show the issue (there are thousands more similar occurrences): 65.55.207.74 - - [13/Dec/2009:14:58:42 +0000] "GET /specimen/235698/ HTTP/1.1" 200 1742 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.207.74 - - [13/Dec/2009:14:59:06 +0000] "GET /specimen/235698/ HTTP/1.0" 200 4259 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.209 - - [13/Dec/2009:15:03:08 +0000] "GET /specimen/250262/ HTTP/1.1" 200 1733 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.106.209 - - [13/Dec/2009:15:03:14 +0000] "GET /specimen/250262/ HTTP/1.0" 200 4164 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" This seems a waste of bandwidth and completely defeats the point of supporting http compression.

Indeed a waste of bandwidth and yes, it defeats the point of supporting HTTP compression.

A Bing representative, Brett Yount said:

could you please mail this information to [email protected] and I will get our crawling team to check it out?

But we have no confirmation from Bing on why this issue is occurring or when it will be fixed. Like I said, just one more oddity to add to MSNBot's crawl behavior.

Forum discussion at WebmasterWorld and Bing Forums.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 18, 2025

Apr 18, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Ruled A Monopoly Again, Heated Volatility, Google ccTLD Change, Ads Safety Report & AI Overviews

Apr 18, 2025 - 8:01 am
Google

Google Made Search Much Faster - But How Much Faster?

Apr 18, 2025 - 7:51 am
Google Ads

New Google Merchant Center Popular Products Report

Apr 18, 2025 - 7:41 am
Google Search Engine Optimization

Google: No Positive SEO Effect From .esports Domain

Apr 18, 2025 - 7:31 am
Google Ads

Google Ads PMax Age Exclusions Rolling Out

Apr 18, 2025 - 7:21 am
Previous Story: When Will Google Fix the Web Design Local Pack Bug?