Clever :) but reason is more prosaic, as time travel is already achieved with special constraints for photons, your body has too much elemental particles, no way to condense such computational power with Moore law, at this point Moore's law suppress your potential to grow and teleportation, it's constraint from top now :)
And a modified version of the double slit experiment demonstrate how to change past from future as well.
But really, is in it beautiful and counterintuitive idea travel across the Universe using simple time travel protocol ?
@vrdad you need to build maps where in Milky Way we can travel this way !!

May be you want to try :)

Written with Copernicus.