Connection Time & HTTP Status Codes Used By GoogleBot For Crawl Efficiency

Oct 2, 2014 - 8:24 am 8 by

GoogleBotThere is nothing worse than knowing Google is having a tough time accessing your content unintentionally. Well, truth is, if you robots.txt or nofollow your site out of Google by accident, I don't feel bad for you. But if your server flakes out on you, then I do feel your pain.

Yesterday, at SMX East, the great Gary Illyes from the Google search quality group, shared two tidbits that you may not have officially heard on-record from Google about crawl efficiency with GoogleBot.

Now, you know that GoogleBot will play nice with your server. If they feel crawling it too hard will hurt the server, they back off. But what signals do they use for determining that? Google has never really shared that information until yesterday.

They use (1) connection time and (2) server status codes.

If Google sees it takes longer and longer to connect to a web page on your domain between GoogleBots hops, it will figure, it should back off a bit or stop crawling. If GoogleBot is served up HTTP server status codes in the 5xx realm, it will also back off a bit or stop crawling. Of course, it will try again later soon but the last thing Google wants to do is take down your site for users.

So if I were you, I'd have reporting configured on (1) connection time and (2) 5xx server status codes.

Forum discussion at Google+.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: April 18, 2025

Apr 18, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Ruled A Monopoly Again, Heated Volatility, Google ccTLD Change, Ads Safety Report & AI Overviews

Apr 18, 2025 - 8:01 am
Google

Google Made Search Much Faster - But How Much Faster?

Apr 18, 2025 - 7:51 am
Google Ads

New Google Merchant Center Popular Products Report

Apr 18, 2025 - 7:41 am
Google Search Engine Optimization

Google: No Positive SEO Effect From .esports Domain

Apr 18, 2025 - 7:31 am
Google Ads

Google Ads PMax Age Exclusions Rolling Out

Apr 18, 2025 - 7:21 am
Previous Story: Google Expands Dynamic Remarketing Ads