Google: Two Common Reasons When A Spike In Crawling Is Bad

Jun 19, 2024 - 7:31 am 1 by

Google Server Explosion

Google's Gary Illyes posted on LinkedIn with two common examples of when a spike in Googlebot activity, crawling, is a bad thing. The short answer is when Googlebot gets to crawling an infinite section of your site (like calendar pages that goes on forever) and when your site is hacked with a ton of new hacked pages.

This is probably very basic SEO stuff for most of you but there might be a reason he is sharing this now.

Gary wrote, "Don't get happy prematurely when search engines unexpectedly start to crawl like crazy from your site." "A sudden increase in crawling can mean good things, sure, but it can also mean something is wrong," he added.

In these cases, he said, "Treat unexpected sharp increases in crawling as a symptom of an issue until you can prove otherwise." Then he joked, "Or, you know, maybe I'm just a hardline pessimist."

Here are two issues that come up way too often when looking at sharp increases in crawling, he wrote:

Infinite Spaces:

The example on infinite spaces that he provided is one of the most common. When "you have a calendar thingie on your site, or an infinitely filterable product listings page. If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time. robots.txt is your friend, use it."

Hacked Content:

Another common example is hacked content. He said, "if a no-good-doer somehow managed to get access to your server's file system or your content management system, they might flood your otherwise dandy site with, well, crap. If your site generally has pages that search users find helpful, crawlers will get excited about these new pages for a time and happily crawl them. https://web.dev/hacked has great resources about what to do in these cases. (tangent: yes, this is more cracking than hacking but apparently the internet is fine with the misnomer)."

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Follow

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: December 10, 2024

Dec 10, 2024 - 10:00 am
Google

Google Crawler Documention Adds HTTP Caching

Dec 10, 2024 - 7:51 am
Google Search Engine Optimization

Google: Sometimes Over Optimization Drift Towards SEO Spam

Dec 10, 2024 - 7:41 am
Google Search Engine Optimization

Google Search Console Insights Removes Google Analytics Data

Dec 10, 2024 - 7:33 am
Bing SEO

Copilot Beta Now Bing Webmaster Tools For 10,000 Users

Dec 10, 2024 - 7:31 am
Google

Google Tests Trending & Popular With Labels In Search Results

Dec 10, 2024 - 7:21 am
Previous Story: Google Tests Two More People Also Ask Results (6 PPA)