We know that Google has said that plus ones can have a positive impact on your rankings but we also know site speed can have a negative impact on rankings.
A WebmasterWorld thread has reports of webmasters pulling the Google Plus One button from their sites because they have been one more element to slow down the page load time of their pages.
One webmaster said:
It was a real pain to add the +1 button and make it fit in layout and get along the other buttons alignment. But what I noticed once I finished spread it all over my site is a horrible slow down in page load.All page loads fast, but the +1 button is the last one in loading and while it does leaves the page like frozen until it does completely. For instance the main navigation menu (pure css) won't show subitems until the +1 button finally appears.
Truth be told, there are ways to speed up the buttons through some tricks. Michael Butler at RustyBrick (my company) posted an article named JavaScript Hover Effects to Speed Up Page Load Time. Basically, you do not load the buttons until a user places their mouse over the button. Before the user does that, the button is simply an image.
One webmaster claimed that his rankings slipped after adding the plus one button and then shortly after removing it, the rankings came back. Of course, there can be other reasons and factors, but this is what he said:
Just thought I'd add that about 10 days after posting that I removed the +1 buttons from the news articles and within 48hrs the traffic to the news section was back to normal levels (up to about 150/200 uniques a day). Two note: 1) adding +1 was the only change I had made to the news section in months and 2) during all that time, traffic to other parts of the site always remained consistent.
The last time we covered a topic like this was with the Facebook like button where a Googler said it will have no impact on your rankings.
Forum discussion at WebmasterWorld.
Update: Hours after I posted this, Google announced a faster Google Plus One button. Clearly this post had nothing to do with them releasing it, I am sure it was in development for a while.