I'm with GM4Him in this. 4 or 6 it should have some decent explanation why we can't take everyone with us. While now we have 6 companions in total there are more incoming, so it seems we will always have some sitting it out in the camp. I can't express how much I'm tired of this reoccurring stupidity in RPGs that you have a batallion, but keep act as a squad. It's the main reason I was happy with DOS2 killing companions, which aren't in the team at the end of 1st Act. But even there the situation could be improved in the 1st Act itself.
Honestly, I am genuinely unhappy with that in DOS2 cause there really isn't any reason why those two other people (or three if you start as a custom) wouldn't have joined you. Even with the godwoken thing, every seemed content to join you and the goal in act 1 was survival and escape.