The problem is with turn-based - changing from 4 to 6 will mean that fight are on average 1.5 times longer, and that's not taking into account having more enemies to balance difficulty.
Why people keep parroting this argument? That's not true at all. A six men party just means that the turn rotation will pass the action more often to one of the characters you are controlling rather than an enemy. Which if anything is something that feels fairly NEEDED in the current build, where way too often you are watching a whole bunch of enemies "Alpha-striking" your party (and focusing on the casters in particular) with very sparse chances to inject your action in the middle. You may even end up cleaning encounters more quickly than with four characters.
Of course, in the long run it MAY be needed to buff the enemy side a bit to balance encounter design. Which can be achieved in a lot of ways that not necessarily pass through "adding more of them".