If you had a teleporter stone in your inventory when entering the castle, an imp historian would have shown up and 'borrowed' them. Continue with the main plot (rescue Zandalor) to get them back.