Many webmasters and SEOs are obsessed with site speed, not just for their users satisfaction but also because Google seems to be obsessed with it as well. But XML sitemaps are not for users, it is for Google, so do those need to load super fast?
The answer is no. They just need to load and how long it takes, doesn't seem to matter too much. As long as they don't time out when Google tries to access it, then you should be okay.
John Mueller of Google was asked this question yesterday on Twitter and he said "as long as it doesn't time out, you should be fine."
Here are the tweets:
As long as it doesn't time out, you should be fine. Maybe cache the latest version to prevent unnecessary server load?
— John ☆.o(≧▽≦)o.☆ (@JohnMu) September 13, 2017
So since XML Sitemaps are not for people, it is okay if they don't load super fast but make sure they do load.
Forum discussion at Twitter.