Back in February 2014, Google began sending a manual action notice for spammy structured markup. Back then I assumed that if you received such a manual action, it meant you lost your rich snippets from showing up in the Google search results until the manual action was revoked.
I did not think it may have meant your rankings would see a negative impact in the Google search results.
Tony Edward wrote a guest article at Search Engine Land about the topic and it seems that some people are thinking that this manual action also may result in your rankings in Google feeling a hit.
I am not sure. It can be, I asked Google about this but currently didn't hear back.
Here is a picture of the manual action, which hasn't changed in context in over a year:
I suspect it depends on how bad the spam is, how often it happens and more.
Have you received one of these manual actions and noticed not just a removal of rich snippets but also a ranking decline?
Forum discussion at Local Search Forums.
Update: Google answered my question in a video saying it only impacts your rich snippets, not rankings.