The camera and the AI pathfinding already struggle with a party of four.
And for that very reason, we can have a bigger party, because the struggle is already a thing and won't change. At least we would have a bigger party to compensate for that. I would be only too happy to accept that trade-off.