Google's John Mueller was asked by a webmaster if he should remove the mobile alternative pages since the site is now responsive. John Mueller said yes, saying "Fewer page types make things easier, so if you can migrate away from m-dot that seems like a good idea to me."
Giving Google fewer alternate copies of your pages, with the exception of AMP and a few other things, is a good idea. It keeps things simpler and easier for Google to process and helps you focus in on less redundant pages on your web site.
Here are those tweets:
Fewer page types make things easier, so if you can migrate away from m-dot that seems like a good idea to me.
— 🍌 John 🍌 (@JohnMu) January 3, 2020
Obviously, when you do this, make sure your 301 redirects from the pages you are removing are pointing to the canonical version. That will not just help Google consolidate any signals from those pages you are removing but direct any users who may have bookmarked those pages to the right place.
Forum discussion at Twitter.