I reported at Search Engine Land that Apple has finally confirmed the rumors of their search engine crawler/spider. They named it AppleBot.
Apple says it is being used "used by products" adding specifically "including Siri and Spotlight Suggestions." But that can be used by more. I'd be shocked if Apple was not looking for a way to be less reliant on Google or Bing or other search engines to power the other search results. Mobile search - truth be told - is still up in the air to be grabbed by the best player and if Apple moves fast and smart enough, they can leverage their marketshare with Siri and potentially get a big win there.
The AppleBot documentation reads:
Applebot is the web crawler for Apple, used by products including Siri and Spotlight Suggestions. It respects customary robots.txt rules and robots meta tags. It originates in the 17.0.0.0 net block.User-agent strings will contain “Applebot” together with additional agent information. For example:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.
If you have questions or concerns, please contact us at: Apple-NOC “at” apple.com
AppleInsider reports that the company is "rapidly-expanding internal search group" potentially to build out a better Spotlight feature for web search.
I guess time will tell but I think it would rock to get another serious web search competitor out there.
Forum discussion at Google+.
