Yes, it happens when you rearrange the party. It looks like for each character it remember where it was last that was not the camp and sends him/her there. The only place where you can rearrange the party is your camp, so you are pretty much stuck with replaced characters talking some long walks to find the action.
