But why do I need to use any energy? Why can't I let go an watch it float off?
How can you let it go if you've not applied any energy? it means you can never pick the brick up in the first place if you do not apply energy to lift it's mass.
That brick is being pushed into th ground. To enable that brick to go into the atmosphere, it requires some kind of energy to do that. You supply that by applyng your energy to lift that brick against that atmospheric force pushing down on it. The fact that you done this means you are holding potential energy and as long as you hold it, that's all it is, until you release your energy by letting go, in which case the atmosphere acts on that brick's density by the air it's compressing by taking up that volume of air that you placed it in.
What is making that brick want to head towards the earth (ie down) rather than any other direction? As air pressure gets lower as you move away from the planet, surely stuff should "fall" upwards if anything?
In your theory, what is so special about the down direction?
The down direction is a direct opposite to energy pushing in the upward direction from the ground.
Think of it like putting coins in a tube from the bottom. The first coins you put in are easy with your energy. As that tube fills up, the pressure upon your finger becomes stronger, meaning you have to use more energy each time you push up.
Eventually when the tube is filled, the coin at the top is resting it's potential energy on top of the other coin and so on and so on until the bottom coin is under the pressure of all the rest against the bottom of the tube which resists it all.
Now imagine that with atmosphere, then you in it. You push it out of the way with your body and that atmosphere pushes down on you, just like it will with a brick.
Now imagine picking up the brick. You compress the air above with every movement because you are placing the mass of that brick into that air and that air fills the void left by where the brick originally was.
If you contnued moving the brick up, it could be action/reaction until you stop. This is when the atmosphere is pushing down against your resistance (hand). Your energy stops it from happening until you leave loose and that potential energy now becomes energy against air resistance.