I get what youre saying, but I think were talking about a slightly different atmospheric model.
In my original post, I assumed people would argue that the same force that causes the earth to accelerate, would also cause the atmosphere to do likewise. So in a sense, there is no inertia felt by the atmosphere, and no force exerted on it by the earth. When this is the case, there is nothing to stop the entire atmosphere diffusing out into space.
If on the other hand we assume that the atmosphere is being "pushed along" by the earth, then yes, I agree we would need infinitely high walls to contain it.