The load of an object causes a point load. If i have cube, i can split it in half. Both ends sill weigh equally much. One end will be close to the pivot and act with less torque, the other will be further from the pivot and act with more torque. If you add the torques together, you get the same result as a point load with the load of the original cube and it's position.

One of the definitions of a center of mass is that the torque caused by gravitational loads from the object itself will be perfectly balanced and equal 0. If you put an object on some structural element and it is in rest, the distributed force will act as a single force concentrated at the center of mass, i.e. a point load.

It's also very important to distinuish point load from pressure. A point load only means that the force will be balanced at that point with the given magnitude. It doesn't literally mean that there is an infinitely small point applying a force. It acts as one, and it's the best tool for making torque calculations.

Let's compare it to gravity - gravity isn't acting on your body as if it's one object, in reality it acts differently on each particle in your body. So really, if you don't calculate the force on each single particle in your body you can't calculate the force of gravity on your body and get an answer that represents reality even closely, right? Just going by your logic.