Google Does Not Support Crawl Delay But Bing Does

Dec 13, 2019 - 7:43 am 0 by

Crawl Delay

Google's John Mueller said on Twitter that Google "never supported crawl-delay in robots.txt." Bing's Frédéric Dubut said on Twitter "Bingbot supports crawl-delay." That is not to say Google does not let you slow the crawl, you can and here is how.

Here are the tweets:

The crawl rate setting is still in the legacy Search Console, but you can still access it over here. You can learn more about Google's crawl rate over here.

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Updates

Google Search Ranking Volatility Heats Up January 29 & 30th

Jan 30, 2025 - 7:51 am
Google Maps

Google Maps Service Area Business Access Request Link

Jan 30, 2025 - 7:41 am
Google Ads

Google: Why 100 Negative Keywords For Google Ads PMax Campaigns

Jan 30, 2025 - 7:31 am
Google Maps

Google Business Profiles Gains Batch Price Editing & Moving Menu Items

Jan 30, 2025 - 7:21 am
Google

Google AI Overviews Go Deeper With Gemini 2.0?

Jan 30, 2025 - 7:11 am
Bing Search

Microsoft Bing Advertising Revenue Up 21%

Jan 30, 2025 - 7:05 am
Previous Story: Google: Speakable Markup Works Outside Of News Content