Google has updated its Optimize your crawl budget page, it said to "improve clarity, terminology consistency, and flow." One thing that is new in the document is that Google said that "Every site starts with the same default, conservative crawl capacity limit."
There are tons of changes to the document, most are not fundamental changes to how Google crawls but just clarifications and improved writing in the document itself.
Google wrote the changes were to "make the documentation easier to understand and ensure technical terms (like crawl capacity limit) are used consistently throughout."
Here are some new lines added:
- "Every site starts with the same default, conservative crawl capacity limit. If there is demand to crawl more and the site remains healthy, Google's systems will automatically adjust this limit over time."
- "For Googlebot, demand varies based on a site's size, update frequency, page quality, and relevance, compared to other sites. The primary factors you can influence are."
- "While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others."
- "Improve loading speed: Optimize your server response times and resources to make pages load faster."
- "Use HTTP caching: Support 304 (Not Modified) HTTP status codes. If a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources."
There are many many changes, here is a before and after comparison:
Forum discussion at X.
Image credit Lizzi Sassman
