Would it make more sense to say that it is the angle that light reflects from, rather than the angle sunlight hits the ground that affects the distortion?
Certainly so, but one depends on the other.
this really depends on the material. If we're talking about a mirror or other highly reflective surface, I'd agree, but not all objects are so smooth. For rough objects like rocks, light seems to reflect at all kinds of angles, due to the multifaceted nature of the rock.
would this not cause a noticeable change in distortion in a daily cycle if distortion was related to this angle?
It would. However, due to no objective reference points, we're not really able to perceive that correctly.
I recall reading about a social experiment (I don't have the source, unfortunately) where adults were put in "magnified" rooms to make them feel smaller, so that they could get an idea of how children feel. Of course, they knew full well that they are entering such a room, so they couldn't possibly be fooled into thinking they're *actually* smaller than they were in the first place, but it still made a decent impression of it. Now, what would happen if they didn't know that every single reference point they might possibly think of changes too?
This only shows that people do notice that something is different. Whether it's the size of the person or the size of the room, the person knows something changed. This seems contrary to your claim that we wouldn't perceive any change. If there is actual distortion, why doesn't my back yard feel smaller at noon than at dusk. If we can't reliably perceive the distortion, how can all the re'ers here manage to notice that Australia is heavily distorted?
Also, if perceived distance is dependent on angle of sunlight hitting the ground, how do we perceive distance at night when there is no sunlight at all?