Google: Crawl Rate & Crawl Demand = Crawl Budget

Jan 17, 2017 - 7:52 am 2 by

Google Crawl Budget

Several weeks ago, Google told us crawl budget is not what we think it is and promised to write a blog post explaining what Google defines as crawl budget. Well, Gary Illyes posted on the Google blog What Crawl Budget Means for Googlebot.

In short, "crawl rate and crawl demand together," Google said is how they "define crawl budget as the number of URLs Googlebot can and wants to crawl."

Crawl rate limit is how much GoogleBot allows itself to crawl your pages in order not to overload your server and do harm. It is the "number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches." Google won't crawl too much to take down your server and it will respect the limit you set in the Google Search Console.

Crawl demand is the demand for Google to crawl your pages, new and old pages. Demand is determined by popularity of your site and URLs and the staleness of the Google systems attempt to prevent URLs from becoming stale in the index. Even if you have more crawl rate left, if demand doesn't require more crawl, GoogleBot won't crawl.

Google said, these two make up crawl budget. "Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl," Google wrote.

What impacts crawl budget?

  • Faceted navigation and session identifiers
  • On-site duplicate content
  • Soft error pages
  • Hacked pages
  • Infinite spaces and proxies
  • Low quality and spam content

Crawl rate is not a ranking factor, said Google. Google wrote, "an increased crawl rate will not necessarily lead to better positions in Search results. Google uses hundreds of signals to rank the results, and while crawling is necessary for being in the results, it's not a ranking signal."

For more details and a lot more FAQs in the blog post.

Forum discussion at Twitter and Google+.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 31, 2025

Mar 31, 2025 - 10:00 am
Google Updates

Google March 2025 Core Update Impact Multiple Poll Results

Mar 31, 2025 - 7:51 am
Google News

Google To Pay $100 Million To Settle Old Ad Lawsuit On Clicks Outside Geo Target

Mar 31, 2025 - 7:41 am
Google Ads

Google To Allow Double Serving Ads In Different Ad Locations On Same Page

Mar 31, 2025 - 7:31 am
Google Search Engine Optimization

Google News Fully Transitioned To Automatically Generated Publication Pages

Mar 31, 2025 - 7:21 am
Google Maps

Google Business Profiles Bug Resurfacing Old Fake Removed Reviews

Mar 31, 2025 - 7:11 am
Previous Story: Tell Google What Technical SEO Topic They Should Write About