So when you are on a skateboard and push against the wall you move away by displacing air ? 
Yep, you move away by displacing air. The reason you can't grasp that is because you're not thinking on the right lines.
Here's a tip. Arms bent and energy applied to the wall, requires a displacement of air, just for that to happen. Once that has happened, then all of the mass has been pushed against the atmosphere, compressing it and causing it to come back around the mass to create equal and opposite reaction to action.
you know you can push against a mass whether it is fixed or not, you could push against a medicine ball when you throw it.
Yep but the only way you can push against a medicine ball and receive a reactionary force back from it, is either when that medicine ball is under atmospheric clamping on the ground which creates a friction on that ground as well as against atmosphere, or your own energy holds that mass of medicine ball, up. You have displaced the amount of atmosphere that the medicine ball repels from its mass and that's the pressure of atmosphere bearing down onto your ball that you have to resist with your energy.
By throwing this medicine ball horizontally, you will have to move it from one filled atmospheric place to another, then another and another, depending on how much energy you put into the throw. All that atmosphere reacts against your ball until you release it - or to put it simpler - when your energy of push does not count anymore.