Google launched the evergreen GoogleBot, a GoogleBot that will be up-to-date with the latest version of Chrome. A GoogleBot that can access all the latest modern web browser features. A GoogleBot with less limitations. I've been updating the original article with more details but if you missed it, here are some new tidbits around this announcement that you may have missed.
User Agent Didn't Change
Despite us seeing the old user agent saying Chrome 41, it is really Chrome 74. Martin Splitt from Google said they didn't want to make too many changes that might hurt SEOs by changing the User Agent name, so they changed GoogleBot but didn't change the User Agent name yet.
Yup. We think it's better to give people who might have a hard-coded the user agent a bit more of a heads up. Expect a blog post on this...
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 8, 2019
How do you determine that it was a Chrome 41 from the logs?
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 8, 2019
But yes, the UA will be updated in the future. More details at https://t.co/mshKgwn8aI
Right, unfortunately the UA is wrong ;-) We wanted to do things step by step and we didn't want to change the UA as people might have hardcoded it somewhere - with enough heads-up, we'll change that!
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 8, 2019
Two Waves Of Crawling
Google still has to do two waves, two passes, of crawling for some JavaScript pages to be fully rendered and processed. Which means, there is still a delay despite GoogleBot getting more modern.
There is still a delay...but more than 1000 new features are supported now!
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 7, 2019
But Google is still working on making it a single pass but has nothing to announce on this yet.
Nothing to announce yet..
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 8, 2019
Less Crawling? Nope.
Martin said this won't result in less crawling of your web pages:
Nope.
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 8, 2019
Many New Features
Like we said before, GoogleBot is able to do a lot more now. Here are some of the features it supports:
Our session on #javascript and #seo at 1:30pm tomorrow will be LIT๐ฅ๐ฅ๐ฅ
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 9, 2019
Come to stage 3 to see what it's all about ๐
Amongst other things, we'll talk:
๐๏ธWeb app architectures
๐คGooglebot
โก๏ธLazy loading
๐Web APIs
๐ฌTesting
๐Crawl budget & more!https://t.co/6CGsiGgg9w pic.twitter.com/b4MnvpSmWt
GoogleBot is Shared
Google's various tools and departments share GoogleBot and while search was updated, not all the tools and other GoogleBots were. So mobile-friendly test, and other tests are not yet updated but will be at some point:
I feel the need to point out that "googlebot" is a piece of shared infrastructure and different teams use it at different settings, in particular, some already have the new rendering engine, others don't. It's not like your Roomba that has only an on/off switch! Oh wait... pic.twitter.com/oqZHtoutmd
โ Gary "้ฏจ็" Illyes (@methode) May 8, 2019
So what's not clear? (Not trying to be smug)
โ Gary "้ฏจ็" Illyes (@methode) May 9, 2019
Googlebot is a service. Teams use it with different settings. Eventually all users will use the new rendering engine, but for now you may run into requests that are still using the old one.
This is hardly worth a blog post
I know and this was not triggered by the UA difference. Someone was doing feature detection dumps on different products like page speed insights, that was the trigger
โ Gary "้ฏจ็" Illyes (@methode) May 9, 2019
New User Agent Spotted
But folks are seeing a new User Agent spotted in the wild but we are not sure if this is related to the evergreen GoogleBot or something else.
@JohnMu @g33konaut Could these hits be due to the new search engine?
โ Lino Uruรฑuela (@Errioxa) May 8, 2019
Googlebot hits using User Agent String Android 9.0.0 Pixel 3 XL, 01-may-2019 pic.twitter.com/t5yZDQTqs3
Great catch Lino. I see that too in my logs. Eager to hear more about this from Google. :) pic.twitter.com/sumpyEZCsr
โ Glenn Gabe (@glenngabe) May 9, 2019
Google is looking into how to respond:
Hi, Lino! Sorry I missed the earlier tweet. Now, that looks like legit Googlebots, but I'll have to dig a little to give you a proper answer! ๐ฌ
โ Martin Splitt @ ๐บ๐ฒ #io19 (@g33konaut) May 9, 2019
So that is all that you might have missed on this news.
Update on this GoogleBot test above:
The assumption about Googlebot you voice might be correct in some cases, but I wouldn't rely on that.
— Martin Splitt @ ๐จ๐ญ๐ก (@g33konaut) May 14, 2019
Alright, so the user agent you saw was an experiment we ran. By the way: we do run experiments on small sets of URLs on a regular basis.
— Martin Splitt @ ๐จ๐ญ๐ก (@g33konaut) May 14, 2019
Forum discussion at Twitter.
