Google has announced the launch of the new crawl stats report and to be honest, it is a huge upgrade from the previous report. This report now gives you data you can use to debug crawl issues and check the health of your site with crawl related issues in Google Search.
The old version just showed you a chart of pages crawled per day, kilobytes downloaded per day, time spent downloading pages by day. Here is a screen shot of the old report:
The new reports really let you dive in and give you the total number of requests grouped by response code, crawled file type, crawl purpose, and Googlebot type. It gives you detailed information on host status, URL examples to show where in your site requests occurred and comprehensive summary for properties with multiple hosts and support for domain properties.
Here are some screen shots:
Grouped reporting:
Multiple host support:
And the details on why Google had issues accessing your site:
Pedro Dias had access to the beta of this for a while and he shared some nice points:
The amount of information you get, and data breakdown of crawls by type of file and status codes, are extremely helpful. pic.twitter.com/7UdnJUy4aM
— Pedro Dias (@pedrodias) November 24, 2020
He added that the "One thing I really like here, is the rendering requests, that are visible under “Page Resource Load”. So you not only see the raw crawl requests, but also the rendering ones."
As Glenn notes, the by purpose is pretty cool as well!
And love the ability to see crawl stats BY PURPOSE (discovery vs. refresh). And then type of Googlebot. Love the information they are including now! https://t.co/ejrNzf68h6 pic.twitter.com/wCW8FAc3IP
— Glenn Gabe (@glenngabe) November 24, 2020
You should be able to access this new report over here and the old report over here.
With the launch of Search Console Crawl stats report, we also published a super comprehensive Help Center article - great read 🤓 https://t.co/NzKWmgXa1b
— Daniel Waisberg (@danielwaisberg) November 24, 2020
You'll find the report in the settings page or just use https://t.co/s9p3Qs5z32 pic.twitter.com/fj62m9LvEX
Forum discussion at Twitter.
