For me the big thing is, with any game where "parties" are even a thing, whatever the party size is, it has to be *narratively* justified. Something about the setting must actually limit the size of groups, or at least they type of group the player's party is. And when no alternate justification is offered, the only justified party size is "the entire group at once, you and all companions".
Baldur's Gate 3 provides no reason whatsover why you can't travel in groups larger than 4. The only "reason" offered is the companions arbitrarily decide they don't want to be a 5th person, and the only conceivable reason for them to think that way is, well, that they are really, really stupid.
Unless Larian wants to come out and update the story to say "The god of numbers and arbitrariness has placed a divine, unbreakable law against travelling groups larger than 4." then they should have balanced the game around the entire team being present at all times from the get go. We shouldn't need a mod to up the limit, and we *definitely* shouldn't need a mod to increase difficulty after upping the limit, it should just be how the vanilla game works.