You need to rescue Zandalor. Head for the Rivertown sewers castle treasure room entrance. Once freed, he will teleport you to the Dwarven Bread inn. After that, let him teleport you somewhere else (you can pick up potions, drop off loot, etc first), clear that smallish area, and you will find the imp historian and get your teleporter stones back.
