To be honest I am really not bothered if the party is six or four but I do remember in 3.5e you could up your party slots is there something like that in 5e?

Cthulhu: FOR THOUSANDS OF YEARS I LAY DORMANT, WHO HAS DISTURBED MY- Oh its you...
Warlock: Greetings my lord-
Cthulhu: LET ME SLEEP-