The idea is to commit and not to change companions after the first act.
4 is a bit tight, if not for everyone, at least for many users. Especially since this game seems to favour large numbers of enemies in battles. But if 5 or 6 were allowed, that would not stop you from only having 4, so why are you posting something like that? So that other people cannot have what they would like? Isn't that a bit selfish?
Balancing is not only an issue of number of characters but also their levels since the world is open, so it's required anyway.