The teleport would have to be scent activated, because everyone "follows your scent back to camp"
.
Maybe you, the PC, just happen to have the exact same smell as the long-ago wizard who set the camp up in the first place. This is just BS that I'm making up as I go along, of course.
