I prefer a 4 member party. Turn based combat (which is very much preferable to real-time w/ pause for a number of reasons) will be slow with bigger parties, especially if you throw animal companions, familiars and summons into the mix. Also, Larian has wisely chosen to opt for fewer companion all over with the trade-off being that each character will be considerably more fleshed out. Finally, a smaller party makes the party selection/character build decisions matter more. Larian has got this.
Combat is slow in BG3 because of its specific combat system, and not because of party size. If a party size of six would make combat more slow, that's not the fault of the party size but rather the fault of the combat system. So don't try to make party size the fall-guy for the flaws of the combat system.
Also, more fleshed-out companions at the cost of fewer companions is not a "wise" tradeoff by any means. If that's the case, why not go down to party size of two? Then we could have even more fleshed-out companions!! It's merely a reflection of your personal preference to have fewer companions. Everything in a videogame is an interconnected mass of tradeoffs and balancing acts. It is perfectly reasonable for other fans to expect both very well fleshed-out companions AND a significant number of companion choices if that's what they want. And the tradeoff to have all that could come in some area, for example by not having all voiced characters or not having romances. There is no automatic or default justification that the tradeoff should have to be fewer companions.
