Tag: robots

Google: Meta Robots Nofollow Is A Hint Also

Sep 12, 2019

Bing: We Never Supported Noindex In Robots.txt

Jul 3, 2019

Google Works To Make Robots Exclusion Protocol A Real Standard

Jul 1, 2019

Google On How To Block Your Staging Server

Aug 15, 2017

Google: Keep The Default CMS Robots.txt Setting

Dec 28, 2016

Unique: AppleBot Will Listen As If It Is GoogleBot

Dec 2, 2015

Google: Don't Make A Dynamically Generated robots.txt

Oct 29, 2015

Google: I'd Avoid Using NoIndex In Robots.txt File

Sep 9, 2015

Google Talks About The Robots.txt Directive For One Whole Hour

Aug 26, 2015

Video Recap of Weekly Search Buzz :: September 12, 2014

Sep 12, 2014

Google Revamps Webmaster Tools Robots.txt Tester Tool

Jul 17, 2014

Google: Can't Crawl Your Robots.txt Then We Stop Crawling Your Site

Jan 3, 2014

Google: Keep Your Robots.txt Simple

Mar 29, 2013

DuckDuckGo Not Respecting Robots.txt Directives?

Nov 27, 2012

Google: Robots.txt Files Must Be Smaller Than 500KB

Jan 30, 2012

BingBot IP Ranges Updated

Nov 16, 2011

Google: Remove The Robots.txt File Completely

Jan 6, 2011

Google Published Detailed Specifications on Robots.txt

Nov 26, 2010

Google: The Robots Meta Tag Is Cumulative (First Come First Serve)

Jul 30, 2010

Bing's MSNBot Crawls Twice, Once For Compressed HTML & Again For Uncompressed

Dec 16, 2009

MSNBot Crawl Delay Doesn't Delay

Nov 11, 2009

Bing's MSNBot Crawl Happy?

Oct 23, 2009

Bing Masking MSNBot Under Mozilla's UserAgent & Reverse IP Fails

Sep 17, 2009

DigitalPoint Founder Upset With MSNBot's Crawl Rate (MSNBot 2.0b)

Aug 3, 2009

MSNBot Clicking on Bing adCenter Search Ads?

Jul 29, 2009

New MSNBot Named adidxbot Causing Trouble

Jul 7, 2009

Tracking Search Bot Activity

Apr 6, 2009