Almost four years ago, Matt Cutts, Google's head of search spam, made a video on how Google ranks the site command results. In short he said, Google may use a form of PageRank to order them, as well as look at how short the URLs are. Matt went as far as to say back then that the site command order output is a "relatively good proxy of the pages that might be kind of interesting."
But today, Google's John Mueller responded in a Google Webmaster Help thread about a webmaster who is concerned that his lower quality pages surface first in a site command, that this person should not worry about it.
John wrote, "the order of results in a site:-query isn't something I'd really worry about."
Here is Matt's video, so you can watch it:
Now, things change at Google, so the site command may have no rhyme or reason these days? Or maybe in this specific case, John is telling the webmaster not to worry.
Forum discussion at Google Webmaster Help.