Things don't fall towards the centre of the Earth.
Then just "down".
You are yet to explain why "air pressure" pushes some things down, but other things up.
You can't even explain why the air stays put rather than just flying off to space.
Pressure is a hydrostatic force. That means it is the same in all directions.
This means an object in an environment at a particular pressure will receive no net force from it and remain stationary.
The sole exception is when it is small and can be effected by Brownian motion.
The other thing which some people may see as an exception is when an object is dividing 2 regions of different pressure. As it has different pressures on it, which exert different forces, it can be accelerated by them.
But an object in Earth's atmosphere has no reason to be accelerated by air pressure.
As for why some things fall and others float, we have gravity, which can be considered bending space time such that the time axis points towards the massive object.
This produces a force proportional to the product of the objects' masses.
This means the acceleration (ignoring other factors like buoyancy and drag), will be the same for any 2 objects paired with Earth, and that is why a steel ball and a aluminium ball fall at the same rate.
Drag is an effect of the particles interacting with the air/gas. This is based upon speed, as it needs to push the air out of the way. In order to do so, it compresses the air in front of it and decompresses the air behind it. This results in a pressure differential that produces a force which acts against the direction of motion, in effect slowing the object down.
This is also relative to the air, so an object can be suspended at a particular height by blowing up from underneath it, and light objects can be lifted by the thermal motion of air. (more complex parts include turbulence and heating the air, but I wont get into that).
It should be noted that this force is proportional to the interaction with air, not the mass of the object.
This is why very light objects with large areas, like a piece of paper or a feather, will often fall (depending on the shape and orientation) much slower than a much denser object which little surface area (like a steel ball), and why aerodynamic objects fall faster than non-aerodynamic objects (however the speed required is not always easily detectable for normal drop heights).
The final point is buoyancy.
This is also due to gravity.
When you put something, like a boat, into some medium/fluid, like water, they both can't occupy the same space.
Both want to fall all the way to the bottom of the potential well (or at least as low as they can go), but they can't both do it.
As the boat forces its way into the water, it needs to displace the water. The more water it displaces the more force is required (as it is effectively holding up that water against gravity). Eventually the force of gravity pulling it down matches the force of displacing and holding up the water, and thus it floats on the surface. It displaces its own mass in water (as its mass determines how much force it gets from gravity, and the mass of the water determines how much force gravity is exerting on it which needs to be overcome).
In some cases (such as for subs, or steel balls), the mass of water it would displace is greater than its own volume.
This means it will displace the water that would otherwise occupy its volume, and that requires less force than the force given to it by gravity. This means it will continue to sink.
However, as it now has to exert force to hold up the water, it will sink at a slower rate (but typically a much more noticable force would be the drag of the water).
The water will also get denser as you go deeper. This means the force required to displace the water grows as you get deeper. For some objects, like subs, they carefully alter their volume, by allowing water in or expelling it, to change the volume occupied and are able to match the density of the water allowing them to just displace the same mass as the amount of water which would occupy their volume, and thus the remain at a particular depth.
The same applies in air. It is just much less noticeable for most objects. Water has a density of 1 g/ml, while air has a density of 0.0012 g/ml.
If you fill a balloon, gravity not only has to pull the balloon down, but also the air inside it (which increases in density slightly due to pressure).
So if you have a balloon which weighs 1.2 g, with 1L of air in it, the air inside will be (ignoring the slight change in density due to pressure), neutrally buoyant. It will be like water in water. If it moves down it has to displace its own mass of air, which takes the same force as gravity is giving it, which means that it provides no net force to accelerate the balloon. This means you only have the force acting on the rubber balloon itself to accelerate the balloon, but it has twice the mass to accelerate and thus will accelerate more slowly.
If instead you fill it with a gas that is lighter than air, like helium, then that helium also has to displace the atmosphere, but the atmosphere it needs to displace is heavier than it, so it would take more force than that given to it by gravity. This means the net force is in an upwards direction. Thus, depending on the mass of helium and balloon (and the pressures involved) it could potentially experience a net force upwards and rise instead of fall.
The other way of considering this is by considering the atmosphere. This atmosphere wants to go down to the bottom of the well as well. But it needs to displace its volume as well.
For most things, like a body or water, it would require much more force to displace the thing that it would get from gravity. As such, it is unable to displace it (and if these things are rigid bodies, then it would need to displace it all. If these things weren't rigid bodies and instead were fluids, then it would need to act over the entire area of the fluid and effectively is trying to push itself down over all of it, which just compresses it a bit rather than displacing any, as displacing some in one spot would just force a different spot to displace it back).
What this means is that the atmosphere will force the helium filled balloon out of the way and fill the void left by it as that will take less force than the force imparted by gravity.
Does that all make sense?
If you managed to get a strong enough balloon to hold in the pressure, and dropped a helium filled balloon in a vacuum, then it would fall.
Does this make sense?
Now then, what is your answer for why things fall or rise?
Remember, if you need to appeal to a force that is dependent on mass, you have effectively brought in gravity.
Pressure is proportional to area.