Nothing aside from math proves anything. There are other sides, but this was slowed greatly by the fact that anyone with a differing view for a long time was marked as a loonie.
Well obviously. If our thirst for knowledge is ever to be satiated, a line has to be drawn where theories are tossed out for their absurdity. If not, we would still be arguing whether the sun is a great ball of fire or not.
But there are also exceptions. It highly depends on the validity of the first theory as opposed to the validity of the other. The evidence for a RE is very strong and with no holes. The theory of gravity, which is awfully similar to electric and magnetic, weak and strong nuclear attraction explains all the observed phenomenon in the sky. FE relies on many disjointed theories to explain each observation as they arise. No gravity, thus UA. No gravity, thus rotating heavens due to celestial gears. Varying accelerations on Earth, thus gravity from celestial objects.
They're branded as loony for a reason...