On July 1, 2024, Reddit updated its robots.txt file to prohibit most search engine and AI crawlers from accessing its content - including blocking Microsoft Bing. Reddit did not block Google Search, which we covered here a few weeks ago.
A Microsoft spokesperson told me last night, "Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site."
Reddit told The Verge, "We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI."
Mark Williams-Cook showed me that Reddit's new content was not coming up in Bing Search, he posted about this on X and then several other media outlets covered it before Microsoft sent me this statement.
Reddit also said, "This is not at all related to our recent partnership with Google." Yes, Google is paying Reddit $60 million annually to be able to access content on its site.
Don't forget that Google has hidden gems algorithm that helped the Reddit gain a lot of traffic from Google. Reddit is seeing a lot of bad PR around Reddit dominating Google Search. I mean, Reddit is dominating Google's search results. Heck, Google even tested special treatment for Reddit not too long ago.
In any event, Bing and many other search engine and AI bots are not permitted to crawl Reddit's site anymore. Google can and does but other engines do not.
Bold move Reddit - but with the Google licensing deal - I guess it isn't too bold of a move.
Wild: Reddit now blocks all major search engines *except* Google from crawling content
— Cyrus SEO (@CyrusShepard) July 25, 2024
This can only increase Google's monopoly power...
... and you can bet regulators will be interested https://t.co/9uGDivBdYc
Timeline:
— Ori Zilbershtein (@OriZilbershtein) July 24, 2024
- Google boosts Reddit in search by over 400%
- Google announces a 60M deal with Reddit right after
- Reddit announces it seeks to go public 48-72 hours after
- Reddit blocks all scraping for competition search engines shortly after
👏Anti 👏Competitive👏Behavior https://t.co/Hzrw0zkjDo
This is part of a disconcerting acceleration of monopolies expanding to further block competition and take from users. Search engines use to provide links to relevant content. Now they spam users with ads intermixed with scraped and content laundered by AI without attribution. https://t.co/xMvzir3DPA
— Tim Sweeney (@TimSweeneyEpic) July 24, 2024
Forum discussion at X.
Update: A bit more from Jordi Ribas of Microsoft on this:
Despite this, Reddit has blocked Bing from crawling their site for search, favoring another search engine and impacting competition from Bing and Bing-powered engines.
— Jordi Ribas (@JordiRib1) July 29, 2024
