The only justification Larian gave for 4 party size that makes sense was to prevent TB combat dragging on too much. As I dislike combat I can relate to that.
On the other hand, more characters may allow more non-combat skill and resolution options to the game, which might be a plus point.
Like every other aspect of the game, more player choice is desirable, but everything comes with a development cost. Maybe this is another option better suited to a mod.