Two more party members is a bigger buff, two additional turns in each round of combat is huge. Each character can also share aggro for the team, etc.

In all honesty, a party of six would give Larian even more leeway to make combat more varied, challenging, etc.