I don't recall whether you get experience or not for getting the teleporter stones back. If you do not take one into the castle, the imp historian will show up the next time you try to use one, or they will stop working until the point you would have gotten them back.
