Just from a phonological point of view...
I think that's the reason why Americans started messing with spelling (Webster and all them blokes) but to be honest there's no point. The English alphabet isn't phonetical and to try and make it so with a few minor changes is pointless, you'd be better off scrapping the whole thing and starting again.
I agree that when you compare colour to words like our, hour and sour or tour or even journey it can look a bit odd but what about pour, your and four? How do Americans spell pour, your and four?
As I said there's no point in mixing with a few words here and there.
I think colour looks better but that's just what I'm used to.
Just as a quick note I pronounce colour as "K
ullah" (Australian) how do other people pronounce it? When thinking about how I pronounce it I don't think either spellings phonologically appropriate.
"u" as in but and cut