In the hangout with John Mueller and Martin Splitt of Google, Martin Splitt said "the two waves of indexing play less and less of a role." He added, "I wouldn't say that two waves of indexing are dead, it's definitely not. I expect eventually rendering crawling and indexing will come closer together."
He also added that a lot more than you think goes through the render phase because it is super cheap. We thought it would be expensive for Google to send pages through this render phase, but this was wrong and turns out to be very cheap - so cheap they send non-JavaScript pages through it anyway.
This is all in the video starting at 32:03 into the video. Here is the video followed by the transcript:
These days, the two waves of indexing play less and less of a role. So basically generally speaking you make see a lot of websites that are not using JavaScript are still going through basically two waves.So here's the thing, pretty much every website when we see them for the first time goes to rendering. So there's no indexing before it hasn't been rendered. And there are certain heuristics that if we see after a while like oh this this page actually the renderer does not diff as much or doesn’t diff, if like it looks the way before... What we do is, we do an HTTP request we get something back write some HTML maybe it's an it's a bare-bones HTML and all it does is load some JavaScript around the JavaScript. Then this HTML that we got from from the original HTTP GET request from the crawl goes into rendering, rendering runs JavaScript, boom a lot of content happened that wasn't there before. So we're like aha okay so this needs to be rendered but there is a heuristic that is.
And the interesting thing is that so what I want to make very very clear right because I talked to the team and I was surprised about this. I thought this is a lot more this is still a lot more like more frequently happening that we are going like oh all right we're gonna skip rendering. It is not as frequently happening anymore.
So like for many many websites even if they do not run JavaScript they might still go through the render phase. Because it doesn't make a difference as much because it's cheaper. It's cheap. It's cheaper than the complexity that we infer. So like there's very very few cases and the internals of that are very complicated and I still haven't fully like grasp what exactly triggers the heuristics.
I wouldn't say that two waves of indexing are dead, it's definitely not. I expect eventually rendering crawling and indexing will come closer together. We're not there yet but I know that teams are looking into it. No plans, no deadlines, no roadmaps, to be announced yet.
I am not sure if this fully explains everything but it does shed more light on this process.
Forum discussion at YouTube Community.
