I'm in the minority on this one... I can't stand having characters standing around in camp doing nothing but waiting for you to return.
It always breaks immersion for me - "Off to fight a small army of goblins! I'll take you 3 and the rest of you lot - just keep on keeping on!"

I hope when you select your party in this you run into the others where they should be in the game, even if it's working against you. Actually, I kind of hope you don't select your party and the members that align with you the most based on your actions stick around while the others leave on their own. It would be weird to help everyone out selflessly and have your selfish vampire and evil cleric friend stick around for the lolz. My favourite Final Fantasy was Final Fantasy 4, where your party would change because the people with you had their own things going on, and the PC game world needs more of that to make NPCs feel less flat.