Generally Google will list a site in its search results with the domain name, not the IP address associated with the domain name. But if you see both and if your site is crawlable via the IP, it is not going to lead to your site being deindexed by Google, said John Mueller.
John said this in a Reddit thread where the user said "my IP address is being indexed for pages and actually ranking." Truth is, that is not so uncommon - I see it a lot.
John Mueller of Google said "having a duplicate via IP address is not great, but wouldn't result in a site disappearing from Search." He actually said something like this back in 2017.
John posted a very long response there about the site, saying his issues are likely not technical SEO related at all. Here is what John wrote, it is a good thread to read:
Impossible to say without looking at the site, but my guess is this is not a technical issue.Looking at the traffic since 2012 is irrelevant - things have changed significantly over the years. If you're on pages 4+ and getting ca 100 visitors/month, you're going to see a lot of fluctuation and those numbers aren't going to be relevant.
The only way a technical issue could result in issues like this would be if you had a noindex on all pages, which I doubt is the case.
Having a duplicate via IP address is not great, but wouldn't result in a site disappearing from Search. Usually it just means that in Analytics you see traffic for both versions and it kinda shifts back & forth as the canonical changes (which it probably wouldn't for most sites anyway - it's pretty common to be accessible via IP address). If the IP address version attracted the bulk of your site's links, that might be something to think about, but honestly, that almost never happens, "nobody" links to IP addresses naturally.
If your site is ranking on pages 4+ and just can't get higher, then my guess is you're trying to compete with something that's quite far below (in terms of quality, content, external support, etc) the main sites being shown. If that's the case, I'd try to figure out the chances that your site will break through to page 1 (if you want significantly larger numbers, that's where you probably need to be), work out what you'd need to do to get there, and compare that to just dropping this project and working on something else to get similar traffic. My *guess* is it would be significantly more efficient to just move on to something else. I realize that's annoying, but realistically if you just move to page 3 and get 10x the traffic for a while, will that really be worth the smaller effort? Instead, aim for something where you can clearly be the top result based on the things that you're good at, the things that are realistically within your reach. If you get that figured out, you can always go back to this site too :)
If you take this and go "I'll show that dude that it's possible" - go for it. I just want to be clear that it's not a matter of technical tweaks or minimal effort, it's going to be a lot of hard work (or, potentially, a lot of "voodoo SEO magic" that gets you a temporary boost -- some SEOs are super-creative, kinda-sneaky, and smart, and can possibly do that, but keep in mind that it won't be a long-term solution).
Forum discussion at Reddit.
