If you have at least one teleporter stone with you entering the castle, both will be 'borrowed'. If you leave both somewhere else, the imp historian will show up the next time you try to use one. If you try dragging a teleporter stone out of your inventory as you enter, the start of the cut scene might cancel the drag, so it would be put back in your inventory, or it might not. In either case, you most likely would not be able to continue using the teleporter stones between entering the castle and clearing the council hall.