Spiders
Older Entries
GoogleBot Getting Tired? Google's Spider Crawling Less Documents
May 1, 2008Google Now Crawling Content Behind Forms
Apr 14, 2008Is Google's AdWords Spider Lowercasing Destination URLs?
Apr 2, 2008Yahoo Slurp, Yahoo Search Crawler, Suffering From ADHD?
Mar 31, 2008MSNBot Again Failing Reverse DNS Test
Mar 11, 2008MSRBot Bot, Microsoft Bot Causing Havoc on Some Servers?
Feb 25, 2008Microsoft Live Search Adds HTTP Compression & Conditional Gets Support to Crawler
Feb 13, 2008Can You Hide Text & Links From Your Users, If Search Engines Won't See?
Jan 22, 2008Google's Remove URLs Feature Removes For 90 Days Only
Jan 22, 2008Wordpress Installation Now Blocking Search Engines?
Jan 22, 2008Microsoft Live Search Continues Referral Spam Tests With MSLIVSOP?
Jan 14, 2008Ask.com Fixes Crawler Issue With Badly-Formed URLs
Jan 4, 2008Ask.com Crawler Inserting Url-Encoded Spaces in URLs Causing 404 Errors?
Dec 31, 2007Microsoft Confirms Reverse DNS Bot Issue & Promises To Resolve
Dec 6, 2007MSNBot Reverse DNS Test Fails Requirements
Dec 5, 2007Live Search's Spam Fake Referrals Were Cloaking Tests, Says Microsoft
Dec 4, 2007Double Check Your Robots.txt: Google Testing New Crawler Directives
Nov 21, 2007Got ASP 2.0? GoogleBot Can Trip a Bug in the Code
Nov 8, 2007Google Says DNS Provider Afraid.org Blocks GoogleBot
Oct 16, 2007GoogleBot Not Sending IF_MODIFIED_SINCE Request?
Oct 9, 2007Googlebot DNS Caching Issue Reportedly Resolved
Sep 19, 2007Possible GoogleBot DNS Issues Causing Indexing Issues at Google.com
Sep 11, 2007Is a Robots.txt File Required for Search Engine Optimization?
Aug 31, 2007Managing the Robots.txt File for Sites Sharing Same Local Files
Aug 30, 2007Becoming a Googlebot for Testing Purposes
Jul 23, 2007Does Googlebot Index Faster When Integrating Google Custom Search?
Jul 16, 2007Yahoo! Supports Robots-Nocontent: Enabling Organic Search Page Section Targeting
May 3, 2007Verify The Bots Accessing Your Site: Is Google.com Sending That GoogleBot?
Mar 7, 2007Google & Search Engines Do Not Mind Bad HTML When Crawling
Dec 25, 2006Popular Categories
Search Video Recaps
Most Recent Articles
Daily Search Forum Recap: December 17, 2024
Dec 17, 2024 - 10:00 am
0
-
by
Barry Schwartz
Google Site Reputation Abuse: Treating Some Sites Within A Site
Dec 17, 2024 - 7:51 am
0
-
by
Barry Schwartz
Google: Disavowing Toxic Links Is A Billable Waste Of Time
Dec 17, 2024 - 7:41 am
0
-
by
Barry Schwartz
Google Adds Faceted Navigation To Help Documentation
Dec 17, 2024 - 7:31 am
0
-
by
Barry Schwartz
New Google Merchant Center Promotion For First Order Discount
Dec 17, 2024 - 7:21 am
1
-
by
Barry Schwartz
OpenAI Opens ChatGPT Search To All Logged In Users
Dec 17, 2024 - 7:11 am
0
-
by
Barry Schwartz