Sometime in early June 2021 some SEOs and webmasters began noticing a spoke in soft 404s reported in Google Search Console. I've seen the topic come up a bunch of times in the forums but it was not as super widespread as the soft 404 errors from April of this year.
John Mueller said in this morning's video that Google has changed how it handles soft 404 detection saying it looks at desktop vs mobile differently but in Search Console only reports what it shows in mobile and not on desktop. So if Google sees your mobile site showing a soft 404 for a specific page, it will label it as such but the desktop site might still show up in the search results for that page. John said this is new and kind of surprised him when he heard it. John said this started sometime about a month ago and they are using examples on how to improve the classifer for soft 404s. He said this at the 21:26 mark in this video:
That being said, there may be an issue with the sensitivity controls Google uses for detecting soft 404s. Again, soft 404 are when Google thinks your page that returns a 200 OK status code should in reality return a 404 page not found status code. A lot of sites serve 200 codes for pages that should return 404s, so Google handles it with soft 404s and reports to you which pages are labeled as such in Search Console.
In any event, there has been a number of complaints in numerous Google Webmaster Help forums over the past few weeks or so with this. Tom Rothwell wrote:
Starting to think that the Soft 404 bug reported in April has returned.Since the start of June we had a slight uptick in soft 404s being reported, in recent days this has significantly spiked. We saw this larger uptick start from the 6th of June 2021 and then roughly every 3 to 4 days at ~2-3 thousand URLs. This has resulted in pages previously ranked #1, #2 for particular keywords being completely removed from Google.
He shared this chart:
Mark Sprenger wrote:
I am currently experiencing an issue for Veneta.com where some of our pages are missing from the organic search results specifically on desktop.On mobile these pages are shown in the search results at their original position. Since 24-06-2021 we see that pages are dropping out of the desktop search results.
Has anyone ever experienced a similar issue before or have any direction to where to look to fix this?
Now, John Mueller of Google was asked about this in a previous Google hangout from the other week at the 18:14 mark into this video where someone asked "Did Google change something with the soft 404 detection?" John Mueller from Google responded "probably." He then said:
I mean, it sounds like it. I think especially with soft 404, one of the kind of approaches we've been taking is to try to automatically learn what is a soft 404 page. And I could imagine that in some situations that leads to situations where we have to kind of like pass something back onto the team so that they can adjust the systems that they have there. So if you can send me some examples, that would be really helpful. I can't guarantee that they'll be able to tune it and make it work for that or that it's specifically on our side, but these, I think, are always useful to have.
Here is the video embed:
Also this morning, it was in another hangout with John at the 21:26 mark where John goes into why mobile vs desktop pages may show different soft 404 statuses. It is interesting to listen to. He said he looked into this issue with one of the sites that posted details in the forum and he said it turned out that Google has a system that tries to recognize soft 404s across desktop and mobile separately. John said sometimes Google detects soft 404s different for the same page on desktop vs mobile and that is why you would see differences between device type. So Google can have different soft 404 statuses across your mobile vs desktop URL. So if you have more examples of this, share them with John so they can improve the classifiers.
Here is the transcript from this morning's video:
I looked into it a bit with the other site that posted the details in the forums and it turns out we have a system that tries to recognize soft 404s across desktop and mobile separately. Essentially what happens is sometimes we see pages especially on desktop that look like a 404 page, so we say this is a soft 404 on desktop, we don’t need to index it. And on mobile it looks like a normal page, so we will actually index it there.That is where you would see this difference between the indexing side, if on desktop or mobile you search, and do a site query for the URL colon and the site shows up normally on one device type and on the other device type you do a site query and it doesn’t show at all, then that is a sign for that device type our systems picked up somehow that this is a soft 404 page.
And the tricky thing here… On the one hand, this was fairly new and I was a bit surprised that we do this. On the other hand, in Search Console we do show soft 404s but we show it for the mobile version. So if on the mobile version everything is okay on your side, then in Search Console it will look like its indexed normally. But for desktop, you won’t be able to see that directly in Search Console.
So that is kind of something new that has started happening maybe a month or so ago, I don’t have the exact time frame there.
Send Google examples of where you see this so Google can improve its soft 404 classifications.
So if you have examples, maybe add it to one of the threads above and maybe Google can debug it on their end?
Forum discussion at Google Webmaster Help forums.
Update: John Mueller made some clarifications on Twitter:
Sometimes either the mobile or the desktop page just doesn't work, and we want to avoid sending users to something which we know won't work. Some sites just don't have all content for all device types, which is fine if that's something they want to do.
— 🍌 John 🍌 (@JohnMu) July 11, 2021
A bit. It doesn't change what we have indexed, but it does change when we show a page. We've had this in various forms for a while though, for example if a page robots mobile crawlers but allows desktop crawlers we might not show it in mobile search. Keeping users happy is hard.
— 🍌 John 🍌 (@JohnMu) July 11, 2021
