Site moves and domain or URL changes can be scary, nerve wracking, dangerous and risky. Even Google's John Mueller said on Twitter that "there's never a guarantee that the site will perform exactly the same on a different domain."
But there are many SEOs and webmasters who have experience doing site migrations and know how to handle it to mitigate the risk. "Moving a website almost always results in temporary fluctuations that settle down fairly soon," John also said.
There is a checklist, Google them, of things you should be aware of when doing site moves from an SEO perspective and beyond. As long as you follow those steps to a T and don't miss anything, you should be fine in the long run. Of course, if you miss something critical, fixing the issue down the road can be hard and complex - although some fixes are not too bad.
I should note, when migrating from HTTP to HTTPS, normally that goes pretty smooth from my experience and is fairly easy. But whole site domain migrations are not always that fun and URL structure changes also are pretty scary.
Here is John's tweet:
Moving a website almost always results in temporary fluctuations that settle down fairly soon, but there's never a guarantee that the site will perform exactly the same on a different domain.
— John ☆.o(≧▽≦)o.☆ (@JohnMu) March 5, 2018
Forum discussion at Twitter.