I'm with GM4Him in this. 4 or 6 it should have some decent explanation why we can't take everyone with us. While now we have 6 companions in total there are more incoming, so it seems we will always have some sitting it out in the camp.
I can't express how much I'm tired of this reoccurring stupidity in RPGs that you have a batallion, but keep act as a squad. It's the main reason I was happy with DOS2 killing companions, which aren't in the team at the end of 1st Act. But even there the situation could be improved in the 1st Act itself.