If you do not have a teleporter stone with you when you enter the castle, they will either stop working until the point you would have gotten them back, or the imp will show up the first time you try to use one and 'borrow' them.