Personally, I prefer a party of 5.

A larger party, i.e. 6+ members, is a bit cumbersome for me to maneuver, while the 4-man party is too limiting. We'll have to wait and see what the modders can cook up, if Larian doesn't change it (and I doubt they will).