You said, "...down is toward the center and not some arbitrary direction outside the globe..." That's a unique objection. I'll give you credit for that. For me down is towards the earth or deeper into The Earth and it is the same for everybody, but for you down is towards the center of The Earth, so for you, one man's down is another man's up or even sideways! That's Weird...
Unique?
Look at the Moon.
Full Moon, if it's easier for you.
See the left edge.
Standing there would make you feel gravity pull to the OUR right.
But YOUR right is not same as our.
Your down s where your weight pulls you, and where you put your legs, to hold your weight in place.
We see your head to the left, but you feel it up, opposite of your weight.
Now see the bottom edge.
Your mate stands there, his weight pull him OUR up.
HIS down is our up.
Another mate stands in the center.
His head is toward us, because he turned his legs to the center of Moon behind him.
OUR behind.
His down.
All three of you are pulled towards center of Moon.
All three of you feel center of Moon as down.
Each from different direction.
When one of them comes to you, his down rotates gradually as he gets closer and closer.
When he reaches you, your and his down are the same.
Now take your big beach ball there, put it between Moon's surface and you, and try to stand on it.
You will feel same type of disballance as here.