I understand your logic and your view point. Long ago, in a far away time, I used to watch science fiction and have similar thoughts.
My statement isn't about conjecture. It is a statement similar to "there is a painting on the wall."
Time is not linear, and people do not all share the same past.
Lets say that you went and changed your past (through certain spiritual practices), and then you went and interacted with friends who knew you from then. Which past do you think they remember? Would it blow your mind if they remembered your new past?
Interesting conjecture, and I myself have thought on similar lines. Like, if I went back in time and placed a huge bet on a football match that I knew the outcome of. What would happen to the future that I had travelled back from?
My conclusion was parallel universes would be created, to accommodate my new actions.
However, these are, as you say, complete conjecture, because, unless you have a completely provable way of changing your past, there is no way of testing your theories, or indeed conjectures.
Nice thought patterns though... :-)
Cg