GoogleBot Not Sending IF_MODIFIED_SINCE Request?

Oct 9, 2007 - 7:46 am 1 by

A WebmasterWorld thread discusses a more detailed issue with how Google's spider, GoogleBot, is crawling some pages. Let me quote the detailed explanation:

I've tried: Checking for the HTTP_IF_MODIFIED_SINCE header and returns "304 Not Modified" if possible.

Problem: Googlebot doesn't always send this header. Even if they already know about a page they doesn't always send the header.

I've tried: Using the expires header to tell google that each page should expire in a month from the request.

Problem: Googlebot keep requesting the pages. They seem to ignore this header.

Brett Tabke, founder of WebmasterWorld, said he noticed these issues as well. jdMorgan, a WebmasterWorld moderator, tried to offer some advice:

Check that the 'expires' header is relative -- Expires after so much time, rather than Expires at a certain time.

You should check your Cache-control server response headers as well.

Is this a Webmaster issue or GoogleBot issue?

Forum discussion at WebmasterWorld.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 31, 2025

Mar 31, 2025 - 10:00 am
Google Updates

Google March 2025 Core Update Impact Multiple Poll Results

Mar 31, 2025 - 7:51 am
Google News

Google To Pay $100 Million To Settle Old Ad Lawsuit On Clicks Outside Geo Target

Mar 31, 2025 - 7:41 am
Google Ads

Google To Allow Double Serving Ads In Different Ad Locations On Same Page

Mar 31, 2025 - 7:31 am
Google Search Engine Optimization

Google News Fully Transitioned To Automatically Generated Publication Pages

Mar 31, 2025 - 7:21 am
Google Maps

Google Business Profiles Bug Resurfacing Old Fake Removed Reviews

Mar 31, 2025 - 7:11 am
Previous Story: Yahoo Reporting Error: Advertisers Reports Showing Charges in Millions