Google: Robots.txt Can Be In Any Content Type

Jun 2, 2020 - 7:55 am 0 by

Lines Google

John Mueller of Google said that your robots.txt content-type can be anything really. He said "any content type is fine," adding "the important part is the content itself."

He also said he has "seen html pages that included valid robots txt lines among a valid html page."

If you are concerned, test it with Google's robots.txt validator.

Oh, the Penguin runner asked this question:

Forum discussion at Twitter.

 

Popular Categories

The Pulse of the search community

Search Video Recaps

 
Video Details More Videos Subscribe to Videos

Most Recent Articles

Google Search Engine Optimization

Google: Embedded Resources Are Cached For A While

Feb 4, 2025 - 7:21 am
Google Ads

Google Ads Prompt To Book A Tagging Appointment

Feb 4, 2025 - 7:11 am
Search Forum Recap

Daily Search Forum Recap: February 3, 2025

Feb 3, 2025 - 10:00 am
Google Search Engine Optimization

February 2025 Google Webmaster Report

Feb 3, 2025 - 7:51 am
Google Ads

Google To Test AI Generated Background Images For Your Shopping Ads

Feb 3, 2025 - 7:41 am
Bing Search

Head of Search at Microsoft Touts Bings Growth & Power Of LLMs

Feb 3, 2025 - 7:31 am
Previous Story: Google & Bing Black Lives Matter Statements