Let's say instead of random aliens coming through the wormhole in the "Avengers" movie, it was actually the Galactic Empire, led by Darth Vader, fully-functional Death Star and all. In the same scenario as the movie -- a fully prepared invasion in force trying to establish a beachhead in Manhattan, opposed by the Avengers -- who do you think would come out on top?
Let's say up front that Vader doesn't have the same kind of personality quirks and psychological imperatives as Loki. Loki is a trickster, delighting in sowing dissent and discord, preferring to take the back-stabbing approach rather than a more confrontational stance. Mostly.
Vader is arguably more aggressive and willing to fight head-to-head. He's led massive military efforts before, better fitting the role of general. Plus, while Loki has a mind-control stick, Vader has a light saber.
In terms of the armies at their disposal, it seems like both the Empire and the Chitauri have about the same level of pilot skill. Namely, they can mostly avoid flying directly into solid objects as long as they're not too distracted. We're not exactly talking the Red Baron here in either case.
Regardless, you have to admit it would be pretty awesome to see Hawkeye shooting TIE fighters out of the sky, massive AT-ATs wading down Wall Street while the Hulk smashes them to bits, Iron Man dodging blaster fire through skyscrapers, Thor calling lightning down on Star Destroyers, Black Widow firing widows' bites at Vader while he blocks them with his light saber, and Captain America wading through waves of Stormtroopers.
I'd still go with the Avengers over the Empire, largely playing out the same way it did against the Chitauri. Except for the ending, which would have Iron Man zipping along the Death Star trench to fire at the portal with his repulsor beams, Thor picking off TIE fighters as he advances. Because apparently the Empire and the Chitauri both buy their giant space ships from the same supplier.