When you exit Iona's dungeon after leaving the castle, you were (or will be) given a quest to rescue Zandalor. After you do that and escape, then let him teleport you somewhere new and clear that smallish area, you will meet the imp again that 'borrowed' your teleporter stones. Head for the Rivertown sewers castle treasure room entrance.
In the meantime, you can get back into the castle without the teleporter stones. If you run in and get stopped, then click inside the castle again immediately after the conversation ends (pausing the game may help), the guards will not be quick enough to stop you a second time.
Welcome to the forum.
