There has been some confusion around page speed recently, espesially around the new Speed Update coming in July. In short, the new one in July, will for the first time, introduce a separate page speed score of some sorts for mobile. Before Google only looked at desktop page speed. Now, Google will look at both desktop and mobile.
So people were thinking Google would use the page speed score from mobile only. That is not true.
I asked John Mueller of Google and he confirmed with me that those searching Google on desktop, Google will use the desktop page speed score of your site and those searching on mobile, Google will use the mobile page speed score of your site.
John said this in a webmaster hangout at the 37:35 mark. I asked "Google is going to use the page speed of which version. Does it depend on if it's being accessed on Google mobile versus Google Desktop?" John responded "Yeah, yeah. It's specific to the mobile search results or the desktop search results."
Here is the video embed:
Either way - based on everything I read from Google, only really really slow pages will be negatively impacted by these page speed algorithms. if you site is a tad faster than another fast page - it won't do much for you in ranking.
Forum discussion at Google+.