But the people that won't enjoy playing with 6 player wouldn't be forced to do so, they could play with 4 or 5 or 3 or whatever and enjoy the game anyway they want, just like we want to enjoy the game with 6, balance or no balance(Cheezus i hate that word, gamers are the most Yin/Yang Zen focused group outside of Daoism. ) .
Not exactly. If party size affects difficulty (no change to exp or combats), I (and the many other players who want a party of 6) can't play a Tactician-level difficulty game with the 6-person party. Because that difficulty would be balanced for a 4-person party (and thus would be too easy). Similarly, someone who wants to play an easy solo playthrough wouldn't be able to, because Story-mode difficulty would also be balanced for a 4-person party. We explicitly cannot enjoy the game anyway we want.
Whereas, if a given difficulty mode remains the same regardless of party size (scaling exp), players can separately select difficulty level and party size, fixing both of the examples I gave above.