GoogleOther: A New Generic Google Crawler To Help Googlebot

Apr 20, 2023 - 7:05 am 0 by

Othergoogle Google Crawlers Googlebot

Google has added a new crawler to its list of Google Crawlers and user agents, this one is named GoogleOther. It is described as a "generic crawler that may be used by various product teams for fetching publicly accessible content from sites."

For example, it may be used for one-off crawls for internal research and development, Google explained. The GoogleOther crawler always obeys robots.txt rules for its user agent token and the global user agent (*), and uses the same IP ranges as Googlebot.

The User agent token is "GoogleOther" and the full user agent string is "GoogleOther."

Gary Illyes from Google explained on LinkedIn that this new crawler will "replace some of Googlebot's other jobs like R&D crawls to free up some crawl capacity for Googlebot."

Gary wrote, "We added a new crawler, GoogleOther to our list of crawlers that ultimately will take some strain off of Googlebot. This is a no-op change for you, but it's interesting nonetheless I reckon. As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot's crawl jobs are only used internally for building the index that's used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot's other jobs like R&D crawls to free up some crawl capacity for Googlebot. The new crawler uses the same infrastructure as Googlebot and so it has the same limitations and features as Googlebot: hostload limitations, robotstxt (though different user agent token), http protocol version, fetch size, you name it. It's basically Googlebot under a different name."

There is no comment on if this crawler may or may not be used for Google Bard.

So if you see this new crawler, GoogleOther, this is what it is about.

I think they should have named it "MiniMeGoogleBot."

Just a clarification from Gary:

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Search Forum Recap

Daily Search Forum Recap: March 28, 2025

Mar 28, 2025 - 10:00 am
Search Video Recaps

Search News Buzz Video Recap: Google Core Update Done, Will Site Recover, AI Mode Expands, Google Ads PMax Search Terms & More

Mar 28, 2025 - 8:01 am
Google Maps

Google Local Knowledge Panel With Promotion Clipable Coupons

Mar 28, 2025 - 7:51 am
Google Maps

Google AI Overviews Itinerary, Lens Tour Guide, Maps Screenshot & Track Prices

Mar 28, 2025 - 7:41 am
Google Ads

Google Tests Contactless Label In Place Of Curbside Pickup

Mar 28, 2025 - 7:31 am
Google Ads

Google Ads WhatsApp Conversation Goals Get Diagnostics & More

Mar 28, 2025 - 7:21 am
Previous Story: Google Maps Lapel Pin Gift