Google has taken a strong stance on mobile SEO, announcing there will be demotions if your site is not mobile friendly or is misconfigured when it comes to being mobile friendly.
Google said the demotion will only impact mobile, smartphone friendly, search results and only impact web pages that are not smartphone friendly or misconfigured when it comes to being smartphone friendly.
Google then documents two of many common smartphone mistakes that webmasters make. The two include:
(1) Faulty redirects such as pointing individual pages on your site to your mobile friendly home page, as opposed to a mobile friendly version of that specific page. When possible, Google may even try to fix that for you.
(2) Serving Smartphone-only errors such as 404ing mobile users, other error pages, incorrectly handling Googlebot-Mobile, and unplayable videos on smartphone devices.
Again, Google is getting serious about mobile SEO. If your site is not built right, it won't rank well in the smartphone search results in Google.
Even more so, Google announced last night at SMX that site speed ranking factors will be coming to mobile pages and from my understanding, it will have it's own speed factor.
Forum discussion at Threadwatch, Google+, Black Hat Forums and Google+ Pierre Far.