Because otherwise the refraction will not be significant enough to cause the effect which is attributed to it. The air you are looking through has to be bending light significantly and in exactly the same way all the time. Hot countries would notice a difference as compared to cold countries, even different seasons would put the horizon (on a clear day) at different locations.