I spotted this thread on Mastodon where both John Mueller and Danny Sullivan of Google was helping internet legend Tim Bray with some SEO issues he was having with Google Search. Yes, he got two Googlers helping him, which is rare, but has happened before.
Tim Bray is famous, well, internet famous, he has a Wikipedia entry that reads, "Timothy William Bray (born June 21, 1955) is a Canadian software developer, environmentalist, political activist and one of the co-authors of the original XML specification. He worked for Amazon Web Services from December 2014 until May 2020 when he quit due to concerns over the terminating of whistleblowers. Previously he has been employed by Google, Sun Microsystems and Digital Equipment Corporation (DEC). Bray has also founded or co-founded several start-ups such as Antarctica Systems."
So when someone internet famous complains about Google - Google notices, well, I think Danny Sullivan at least notices.
Tim wrote:
I have reported before that Google is losing its memory - see https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is-losing-its-memoryIt's getting worse. I was looking for a blog piece earlier this year in which I mentioned a bike accident I'd had, and remembered that I'd been wearing a Bontrager helmet (recommended BTW) so I searched for “bontrager” via
bontrager site:tbray.org
and Google can’t find it. DuckDuckGo and Bing can with the exact same string.
Search used to matter to Google.
So Danny and John stepped in to help with some SEO issues with the site and also saying they will send the feedback to the right teams at Google Search.
Danny first came in and noticed it and wrote, "If you have an example you want to share in the future, happy to look. That’s different from us not having indexed a page — @timbray I see @johnmu gave you a reply on what might be tripping us up in this case"
John then dug in a bit and found some issues, he said:
Hi Tim, I work with the search folks at Google. I took a quick look here, and will pass a note on internally.To cut to the chase, what happened here is we indexed https://www.tbray.org/ongoing/goto-potd/ from your site while it was redirecting to your E-Bike article, so we indexed that content under the "potd" URL. Then, the contents for "potd" changed (I assume this is on design), and we indexed that, and lost the E-bike content.
There are a few ways to fix this:
- Google could just figure it out and deal with it on their own. I passed this on, so that we can improve the systems, but it's a weird edge-case, imo.
- block the "potd" URL with robots.txt so that it can't get picked up by search engines.
- use link-rel-canonical annotations on the individual pages so that Google is more likely to pick those URLs.
If you'd like examples of the last two, happy to dig some up.
He then had a bit of a back and forth on some SEO questions with Tim, all useful for us to review as well. Here are some of those responses:
We generally pick up the robots.txt file about once a day (it depends a bit on the site, but since it's a static file, we try to cache it to reduce the load on the server). My guess is by "tomorrow" (depending on timezone :-)), it will stop crawling that.With indexing of that URL vs the article, I suspect it'll take longer for the systems to realize it needs to re-evaluate the situation (I'm guessing a week or so, but it's impossible to say).
It can get tricky when our systems think they've already seen the content, just on another URL (there's so much duplication on the web).
In general though, it's rare that we'd index everything from a website. This can result in even pages closely linked from the homepage not getting indexed. I don't want to set the expectation that a technically clean site will always have everything in search, because it's almost never the case.
Danny also answered some general questions about how Google Search works in that thread:
No, we don't deprioritize older content (nor was this post from April 2022 "old"). We try to show as much useful content as we can. In this particular case, it's likely indexing tripped up for a technical reason.
We index pages. Old and new. We also rank pages. Old and new. Sometimes, it can help to rank pages that are fresher, such as if there's a trending issue going on. I think that generally makes sense. This explains more about how #Google #search makes use of freshness in ranking.
I do believe when someone who is internet famous, is followed more and respected more online, complains about Google Search, it does get Google's attention more than a normal person. But at the same time, it makes sense, because others also follow them more and Google, at least from a PR perspective, wants to jump on those concerns before someone like me writes it up. Plus, I think Danny just follows Tim, so he probably saw it that way anyway.
But yes, being internet famous doesn't hurt when it comes to getting support from Google with SEO questions. It doesn't mean Google will press a button to magically make Tim's site rank better, by the way...
Forum discussion at Mastodon.
Update: Tim confirmed the changes implemented fixed the issue. He said on Mastodon, "Yay, it's fixed, as of now. Everything in my blog back to 2003 seems to be fully searchable. Thanks once again for the help. I updated the blog piece and will post here and on Twitter."
