i'm sure it's been mentioned but one downside of a party of 6 is that you'd have to add more enemies to keep balance, making combat take even longer.
In a RTWP game this is a non-issue because combat is so fast to the point you need to add trash encounters. But in turned based, it can make things feel really sluggish. I already feel some encounters in BG3 are way too slow when too many creatures are involved (i.e. Shattered Sanctuary).