A recent small study by Joy Hawkins showed that if you remove the address from your Google Business Profile listings your rankings in the local pack and in Google Maps will significantly suffer.
The kicker is that Google's own advice and guidelines specifically state, "Important: If you don't serve customers at your business address, do not enter an address under the "Info" tab in Business Profile Manager. Leave the "business location" field blank."
So Joy did that for one of her clients and the rankings dropped off almost completely. So Joy added the address back and the rankings returned. Here is a chart showing the rankings before and after the address was removed and then again, when the address was added back.
She tried this again for another client to be sure and the same thing happened, she was able to replicate the results.
The address is editable in your Business Profile listing, here. Removing the address or sliding the toggle to not show to customers, is the same thing, Joy explained.
For the one I outlined in the article, the address was listed, we slid this thing to off to remove it, left it hidden/off for a month, then added it back. We then went back to 2 other listings that were SABs (no address) and added the addresses and saw the same pattern.
— Joy Hawkins (@JoyanneHawkins) June 16, 2023
But again, if you remove the address, then you will likely see a huge drop in leads from Google Search and Google Maps.
Here is a video of Joy explaining this, if you don't want to read the blog post:
Forum discussion Twitter.