It's not really and argument against it, just and observation. The game as it is now would become too easy and the inventory management even more painful. Imagine manually jumping with 6 characters EVERYTIME, God no.
Which is precisely why I already said several times that revamping controls is basically a prerequisite to expanding the party. And a necessity regardless of it, frankly. Because "it's not TOO painful if you limit yourself to 4 characters" is a terrible argument in favor of a comically bad control scheme (and a painfully inadequate inventory management, too).
Party control in Baldur's Gate 3 is a complete mess that begs to be addressed. SAY NO TO THE TOILET CHAIN