Surplus party members will just hang out at camp and will move with the camp as far as I am aware. I don't believe you have to leave anyone behind.