If you complete the majority of quests in Rivertown and Verdistis and clear the Rivertown sewers (some people also suggest doing part of the Dark Forest) before entering the castle, it will minimize the time you'll need to get the teleporter stones back.
