This is kind of how I feel about it. However fine tuned the encounters need to be for a certain party size, the bigger draw for having more party members is seeing how they'd interact with each other and the the world.
I even wouldn't be totally opposed to a slightly unrealistic system where every one in the roster was 'present' for dialogues and banter, but only a few were 'on deck' for combat. You might even go full JRPG and have a system where you swap people in and out during combat. Maybe that's a bit too much.
I think a problem with a lot of RPGs is how you could find a good party composition and never really veer from that for the whole campaign. I'll give BG:3 some kudos, a lot of encounters become less difficult if you match the right people to the challenge, it's just a bit of a hassle swapping them in.
