I've had this bug as well. As best as I can tell, the secret markers vanish after you are first transported to the End of Time.