Continue with the main plot to get the teleporter stones back. After you leave the castle and go through another area, you'll get a quest to rescue Zandalor; do that and let him teleport you somewhere to find the imp.