Yes, you said
"Visual size is determined by the angular difference between the incoming rays of light."
Notice the difference?
Visual size, not the size of light.
They are quite different.
But the incoming rays of light is just geometry.
Yes, it is just geometry which determines how large an angle that object subtends and thus its visual size.
It is also what determines the direction to an object.
And the 2 are fundamentally related.
If the object is in a particular direction, you look in that direction to see it (or more technically if the rays of light are coming from a particular direction, you look in that direction).
If the bottom and top of the object are in slightly different directions, then you look in slightly different directions to see it, and that difference is its angular size.
This is what actually makes sense.
The visual size of an object is dependent upon the difference in direction to the various parts of it.
If there is a small difference in direction, the object has a small visual size. If there is a large difference in direction the object has a large visual size.
The only limitations to this are when it becomes too small to resolve due to the diffraction of light as it enters your eyes, at which point it will appear as an unresolved point, or when there is very significant glare, but the angular size when you remove the glare (such as by using a filter) remains as it should be.
The brightness of an object (or another such nonsense) has no effect on the visual size (other than due to glare). An easy way to confirm this is right in front of your.
Turn off your monitor and look at it.
Then turn it on with a white image.
The visual remains the same.
The amount of light coming from the screen has no effect.
The essence of light is the wave.
So you reject the wave particle duality of light?
The designers of solar panels, LEDs and so much more would very strongly disagree with you.