This is just a random tip, make sure not to misconfigure Cloudflare, your content delivery network and/or your hosting server. I see way too many cases of this popping up and causing such a pain for SEOs to fix. All of a sudden, you might notice your rankings drop off a cliff on Google Search and your pages begin to drop out of the Google index. It is likely because you are blocking Google from crawling your site.
There are tons of settings in Cloudflare and similar tools, many of them aim to protect your site from AI bots and rouge crawlers. But sometimes, if you don't read or understand them, you can end up blocking search engines and AI crawlers that you did not intend to block. So just be careful.
Jonathan Bird shared on example on LinkedIn recently, he said:
Please be careful with CloudFlare configuration. I just had to fix a website who's traffic tanked for 2 weeks. Their Managed IT Provider (won't be naming them as it's an easy mistake to make, just catastrophic) toggled on CloudFlare's crawl control to stop all bots crawling the site.This completely blocked Google from crawling the site. This meant it broke their Google Ads (who happily kept taking their money), all Merchant Centre Listings disappeared, and Google removed from them search organic traffic. Absolutely tanked their business for 2 weeks, and only just starting to recover now.
Here is his screenshot:
Brodie Clark also posted an example of a different site doing this on LinkedIn, he said:
This was an SEO disaster... while it may look like the result of a core update or even a spam update, it wasn't.It was a result of bots heavily impacting the servers of an online marketplace (this is just one of many domains), and the team needed to put in place bot access restrictions at the firewall level that destroyed SEO just to keep the site up.
Here are some of those screenshots showing the damage:
This is more and more common these days but this isn't new. Even 20+ years ago, I cannot tell you how many examples of sites blocking Google or other search engines from crawling, either by Apache rules, robots.txt misconfiguration, and so on. But now with all the various tools and configuration options, it can be even more confusing...
So be careful.
Forum discussion at LinkedIn.
