You haven't advanced in time nor went back in time. Everything you do, is done in your present time; always.
You may walk forward to drop your pebbles, then pick up those pebbles as you about turn.
All you're doing is retracing your steps in present time. You pick up your very first pebble you put down as you come back to the start. You do this in your present.
I agree, everything exists in my present: however, what is my present may not have always been the same as yours. A month ago, by my perspective, my present was what you'd call your future.
Time travel is travel through the time of another's reference frame. From my perspective, I simply am. If I take a certain event however, I will clearly exist in the past or futue or present, relative to that event. That is all time travel is. If we choose an event that occurred at the same time as me entering my time machine, while that event would once have been by present, it is no longer.
All time travel is, is movement in a new direction.
It is interesting you think of it in those terms, however. One of the forerunners to modern time travel was the D-mail: essentially, a way to send emails and texts to a device in the chronological past. The only way to tell they had been sent, was to see that the phone responsible has no record of actually sending the mail. Remembering the past worldline would, obviously, be impossible (at least, theoretically. A friend of my father's designed the D-mail before I was born, he claimed to be able to remember the past worldline. Then again, he always was rather pretentious, it may have been a lie).
If you believe him, however, when the message was sent, he didn't remember anything of the new worldline. He was simply thrust into the same point in time, resulting from whatever past was altered by the message. However, there was no true time travel involved: essentially, only the present was altered.
Many of your objections do seem based in a misunderstanding. I am the D-mail, in this case. I am irrelevant: what matters is what I call my present, your future. Everything there will shift: the present will change. The details here are what leads up to and causes that to happen, but depending on your reference frame, only one of two things is happening (relative to the worldline). I am someone who came out of nowhere, and am simply part of events: or everything that happens now is no more than a memory, and my present, your future, has simply shifted worldlines.
I hope this was clear. The theory of time travel is still not completely understood, but this is what I know.