If you don't bring a teleporter with you into the castle, the imp should show up the first time you try using them after getting out, or they stop working until the point you would have gotten them back.