Actually I did an experiment in a new game where I did not go to the End of Time at all.

Marker still vanished after a while.