Try walking around the duke's room close to the walls, checking the mini map. Sometime he hides, and only shows up when you get close. The first time I entered his room he was apparently not there, so I decided to loot his chest; when I was just about there he stepped away from the wall into sight.
It is an imp historian who shows up to 'borrow' your teleporter stones. If you do not bring one with you into the castle, he will either show up the next time you try to use them, or they will stop working until the point you would have gotten them back.
It is part of the main plot that you fail to meet Zandalor at the inn. There are several quests that are designed to fail, as well as quests where two opposing sides ask for your help, and you must choose which quest to complete and which to fail.
