After you leave the castle there will be a bit of a detour, then you will get a quest to rescue Zandalor. Do so and let him teleport you somewhere to get your teleporter stones back.