That could explain why the camp is not on the world map.
But there would still be the issue of how the party can teleport to camp and come back to where they were when not near a portal.
Or why we're never surprised in camp by non-party creatures (especially enemies), except when the plot requires it (camp followers, celebration, Scratch, ...).
