Ever since Google announced page speed is a ranking factor, SEOs have been blaming their rankings on their site speed. Google is trying to make webmasters conscious of making fast loading pages but with anything Google does, when they provide a testing tool webmasters and SEOs can get a bit obsessive.
Page speed is important but how important? A WebmasterWorld thread has one SEO/webmaster blaming the speed of his pages for his loss in rankings. But the thing is, you won't be penalized or kicked out of the index for having a slow site. You might not rank as high as an site on the same level as you that might be a second faster.
From what I understand, speed is a very very minor ranking factor. So relax.
As Tedster points out in the thread, you should be more concerned with speed in terms of your users, not GoogleBot. Tedster said:
I'd say that visitors are punishing slow sites much more than Google's algorithm ever will. So improve your speed for you visitors' sake and your business will improve. Even more, when visitors like your site, they are more likely to help you promote it on their own and your links and mentions around the web will grow, too, and that REALLY helps.
Got that? Good.
Forum discussion at WebmasterWorld.