I'm Dunmer and decided to join the Empire not really because of the racism involved with Ulfric's cause, but the futility and counterproductivity of his cause. The empire isn't really oppressing the country of Skyrim at all, and only serves as a layer of protection and provides a connection for economic welfare. Yes, the suppression of their worship of Talos is unfortunate, but as has been mentioned: the treaty betwean the Thalmor and Empire is TEMPORARY and the Empire really doesn't like it much more than the rebels of Skyrim. Any sensible Nord I've talked to in Skyrim actually feels this way as well. Rikke is a perfect example, though some might argue her being brainwashed or whatever.
Also, if one wants to be technical, the Empire was founded by a Dragonborn Nord--the very man/god the Nords wish to worship--and to their perspective should be the rightful ruling body over the lands.
I'm hoping the next Elder Scrolls involves the empire crushing the Thalmor! Because fuck them! Unless I decide to roll an Altmer, in which case fuck the empire, we are the rightful rulers of Tamriel!