Yesterday, Google's Matt Cutts posted a video as a "public service announcement" asking SEOs and webmasters who are blocking GoogleBot from accessing their CSS and JavaScript files to stop blocking Google from accessing those files.
Matt Cutts said it will make for a better crawl and give Google get a better idea of what is going on with your site and return better results to the user. Here is the video:
Going forward, will you block GoogleBot from accessing your CSS and JavaScript files? Take my poll:
Here is the transcript of this video:
0:02 Hi everybody.
0:03 Matt Cutts here.
0:04 I just have a short public service announcement rather
0:07 than a question today.
0:08 And the PSA is this.
0:10 If you block Googlebot from crawling JavaScript or CSS,
0:15 please take a few minutes and just take that out of the
0:17 robots.txt.
0:18 And let us crawl the JavaScript, let us crawl the
0:20 CSS, and get a better idea of what's going on on the page.
0:24 A lot of people block it because they think, this is
0:26 going to be resources that I don't want to have the
0:29 bandwidth or something.
0:30 But Googlebot is pretty smart about not crawling stuff too
0:33 fast. And a lot of people will do things like, they'll check
0:38 for flash, but then they're including some JavaScript.
0:41 And they don't realize, including that Javascript, the
0:44 Javascript is blocked.
0:45 And so we're not able to crawl the site as effectively as we
0:48 would like.
0:49 In addition, Google is getting better at processing
0:52 JavaScript.
0:52 It's getting better at things like looking at CSS to figure
0:55 out what's important on the page.
0:58 So if you do block Googlebot, I would ask, please take a
1:01 little bit of time, go ahead and remove those blocks from
1:03 the robots.txt, so you can let Googlebot in, get a better
1:06 idea of what's going on with your site, get a better idea
1:09 of what's going on with your page.
1:10 And then that just helps everybody, in terms of if we
1:13 can find the best search results, we'll be able to
1:15 return them higher to users.
1:16 So thanks if you can take the chance.
1:18 I know it's kind of a common idiom for people to just say,
1:21 I'm going to block JavaScript and CSS.
1:23 But you don't need to do that now.
1:25 So please, in fact, actively let Googlebot crawl things
1:29 like Javascript and CSS if you can.
1:31 Thanks.
Personally, I rarely block these files from Google or anyone else.
Forum discussion at WebmasterWorld.
