Party of 6 does not always mean more ennemies. This is a wrong argument.
All else equal, if you allow parties of 6 right now, the game becomes much easier. You could balance this through split XP as mrfuji suggested, or by adding more enemies / stronger enemies.