There is a massive, occasionally ill-tempered, thread dating from early access discussing larger party sizes over at https://forums.larian.com/ubbthreads.php?ubb=showflat&Number=672266.

I'll not re-hash arguments from there, but for me personally, four party members supplemented by the occasional guest and summon feels like the sweet spot and I'm happy with it. But as the megathread shows, others vehemently disagree.

"You may call it 'nonsense' if you like, but I've heard nonsense, compared with which that would be as sensible as a dictionary!"