Google doesn't want to list the same content in the search results for a single query, so they do a lot of duplication detection to fold together URLs and content so they don't show a messy set of search results. One would assume that Google needs to first crawl the pages to see the content and meta information before folding and consolidating those signals into a single page. But no, Google's John Mueller said often they can do this before they even crawl the site.
Google's John Mueller said in a webmaster hangout this morning at the 24 second mark that "we also do that essentially before crawling." Meaning, Google also can fold content together before they even crawl the site. "Where we look at the the URLs that we see and based on the information that we have from the past we think well probably these URLs could end up being the same and then we'll fold them together," John Mueller added.
He said they can do that "when we can recognize clear patterns." "Especially within a website where we can see well everything in this subdirectory here is the same thing as in this sub domain because of the way that the hosting is set up so we can just kind of like blindly assume that these URLs are the same without actually looking at them," he added.
I assume some of this is based on canonicals, redirects and basic CMS structures that show clear patterns and signals that Google can pick up on faster than crawling the whole site.
Here is the video embed:
Here is the transcript, although the video was a bit choppy for me:
We we try to do that when we look at the content. That's kind of like after indexing we've seen that these two pages are the same or almost the same, then we can fold them together.But we also do that essentially before crawling. Where we look at the the URLs that we see and based on the information that we have from the past we think well probably these URLs could end up being the same and then we'll fold them together.
That usually makes sense when we can recognize clear patterns. Especially within a website where we can see well everything in this subdirectory here is the same thing as in this sub domain because of the way that the hosting is set up so we can just kind of like blindly assume that these URLs are the same without actually looking at them.
Forum discussion at YouTube.
