I found an interesting tidbit while reading a somewhat detailed thread at Google Groups. The scenario is as follows. You have blocked Googlebot from accessing your site for a 6 month period or so. Then you want to welcome Googlebot back into your site by removing the disallow from your robots.txt file. Will Googlebot bite? If so, how long will it take?
Yea, you ditched Googlebot and now you realize what a good friend Googlebot can be for you. So now you want to become friends with little old Googlebot again. Truth be told, the longer you ditched Googlebot, possibly the longer Googlebot will accept your renewed friendship. Just like real friends. Well, not really.
It is just a matter of how many times Google will recheck your robots.txt file. If you blocked Googlebot in your robots.txt recently, then I would expect Googlebot to check the robots.txt more often. If it is six month or longer, than likely less frequently. It just makes sense from an efficiency standpoint on both Google's side and for your server.
Susan Moskwa of Google has a great analogy in the thread, so let me quote her:
I sometimes use a telephone metaphor to explain this--imagine you called someone and their answering machine said "I can't come to the phone right now, please don't call me." If you called back an hour later and got the same message, and then called back the next day and got the same message, and then called back the next week and got the same message, you'd probably call less and less frequently, assuming the response would be the same, right?The same is true of Googlebot; if your site consistently sends the message "Don't crawl me," we may wait longer and longer between the times when we attempt to crawl it again.
So what can you do to encourage Googlebot to come back quicker? Send it flowers and chocolate. Just kidding! But you do want to entice Googlebot to come back. How so? Submit a Sitemap file to Google, get more links, get buzz out about your site, write more content and so on. As Susan said, "once it starts being 'active' in the online ecosystem, it'll naturally end up in search results and start ranking accordingly."
I had fun with this post because it is pretty late now, been up really long with little sleep. In fact, I feel like I am going to fall asleep while typing this. In any event, hope you enjoyed this little tidbit, it will be posted in the AM.
Forum discussion at Google Groups.
