Sites hosted on SiteGround last week found themselves not being crawled by Googlebot, Google's crawler. The issue seemed to have been an DNS issue between the provider's partners (AWS) and Google according to the hosting company. The DNS issue was resolved after a few days but with DNS, things take time to update and now sites are starting to be crawled again.
As I reported at Search Engine Land last week, Matt Tutt was one of the first to notice this issue - where it started up on Monday, November 8th. SiteGround said they fixed the issue on Friday, November 12th.
Here are the tweets from SiteGround:
We have escalated the issue to Google and we are working to troubleshoot and identify the cause of the problem. We will keep you updated once there's more information or the problem is fixed.
— SiteGround (@SiteGround) November 10, 2021
Status Update: We are glad to inform you that we have implemented a fix for the Google bot crawling issue experienced by some sites. Websites are already being crawled successfully. Please allow a few hours for the DNS changes to take effect. Thank you for your patience!
— SiteGround (@SiteGround) November 12, 2021
But was it fully fixed?
Why's that - am I missing something? They just said they'd traced it down to that specific issue, no fix made. I just tested one of my sites and got the below response. pic.twitter.com/Mh42Eci1QC
— Matt Tutt (@MattTutt1) November 12, 2021
There were more complaints after the 12th and then SiteGround said they applied additional fixes on the 13th:
We have applied a fix last night and sites are gradually being re-crawled. As for the CPU inquiry, please DM us your domain so we can investigate and advise further: https://t.co/zMYnoTzgn7
— SiteGround (@SiteGround) November 13, 2021
It seems since the 13th, SiteGround customers are more silent about the issue, which may mean the issue is now resolved?
The ongoing Google Webmaster Help thread where Caio Barros from Google has been helping provide updates to those impacted, seems to have quieted down as well. It is an interesting thread to scan through, if you want to see it.
John Mueller of Google also provided some information on how Google deals with these DNS issues, in short - once the issue is resolved, things should go back to normal over time.
First off, this happens every now and then, it can happen to us, it has happened to other big services ( https://t.co/NjQeVR0Hrr ). Short outages sometimes go unnoticed, since DNS can be cached.
— 🧀 John 🧀 (@JohnMu) November 12, 2021
Googlebot generally does DNS lookups on the same servers, so our newly published list of IP addresses can help with checking & allow-listing. You can test using the mobile-friendly testing tool & URL inspection in Search Console.https://t.co/32pLcCkiXz
— 🧀 John 🧀 (@JohnMu) November 12, 2021
Because DNS server changes take time to propagate, unless you know it will persist for more than a day, it's usually not worthwhile to move to a different infrastructure. Often it's easiest to keep DNS with hosting, so that any IP address changes can be updated automatically.
— 🧀 John 🧀 (@JohnMu) November 12, 2021
There are generally no lasting effects from temporary outages like these. Technical issues come & go, we have to do our best to make sure users can find their way to your wonderful sites through search results.
— 🧀 John 🧀 (@JohnMu) November 12, 2021
Because of the way Search Console aggregates information about URLs via indexing, the error counts likely won't go down to zero immediately (even if the errors are now gone) -- that's fine and just a reporting issue.
— 🧀 John 🧀 (@JohnMu) November 12, 2021
If you're still here and wondering how to completely prevent this in the future: it's hard. Mainstream hosting providers do an amazing job of keeping things running, and it's extremely rare for me to hear of outages.
— 🧀 John 🧀 (@JohnMu) November 12, 2021
Send them some chocolate (not cheese):
What would I do if a site I cared about were affected? I'd probably be angry & send the folks involved some chocolate. They've been doing awesome work, and I'm sure they'll coordinate with infrastructure folks to make it work even better. 🍫
— 🧀 John 🧀 (@JohnMu) November 12, 2021
SiteGround is not a small hosting company, Wikipedia says the company hosts about 2 million domain names.
To be clear, I don't think this was a big AWS issue, if it was, many more sites would have been impacted. It seemed something specific to SiteGround and how they used AWS?
Forum discussion at Twitter and Google Webmaster Help.
