Google Robots.txt Parser Updated In GitHub

May 23, 2024 - 7:51 am 1 by
Filed Under Google

Google Spider Web

Google has updated its open source robots.txt parser code on GitHub the other day. Gary Illyes from Google pushed the update yesterday morning to the repository there. Google originally released the parser to the world back in 2019.

Gary Illyes explained on LinkedIn that this updated parser has been being used by Google for a while now, but now Google has released that update to GitHub.

Gary wrote, "The release introduces new capabilities in the parser class that allows you to export parsing information about the passed in robotstxt body, and adds a new library to access that information. This new library has been used by Google Search Console for many moons now (in conjunction with the Java port) and so far we haven't encountered issues with it; if you do, file an issue on GitHub!"

When Google initially released this parser, Google wrote they "open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense."

Forum discussion at LinkedIn.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
- YouTube
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google News

Google Ad Revenue Up 8.5%, Overall Revenue Up 12% & AI Overviews Served To 1.5B+ Users Monthly

Apr 24, 2025 - 4:34 pm
Search Forum Recap

Daily Search Forum Recap: April 24, 2025

Apr 24, 2025 - 10:00 am
Google Updates

Google Search Ranking Volatility Heats Up April 22-23

Apr 24, 2025 - 7:55 am
Google AdSense

Google AdSense Policy Center Adds 3 Policy Issues, Filters & Review Process

Apr 24, 2025 - 7:51 am
Google Search Engine Optimization

Google Merchant Center EnergyConsumptionDetails Replaced By certificationIdentification But Backwards Compatible

Apr 24, 2025 - 7:41 am
Google Maps

Google Business Profiles Video Interview On Verifications Must Knows

Apr 24, 2025 - 7:31 am
Previous Story: Google Ads Referral / Affiliate Program