You need to rescue Zandalor (the quest you got exiting Iona's dungeon). Head for the Rivertown sewers castle treasure room entrance. Once freed, Zandalor will teleport you to the Dwarven Bread inn. After that, let him teleport you somewhere else (you can pick up potions, drop off loot, etc first), clear that smallish area, and you will find the imp historian and get your teleporter stones back.
