So long as the difficulty is exactly the same, which I'm afraid won't be possible. And that it's a hard toggle, and no way to recruit past 4 without it on.
Because if you can, you have to. That's just how it works sometimes.
Why do you think it's impossible for the difficulty to be exactly the same? Or at least similar enough that a single individual wouldn't be able to notice it?
There are so many knobs that can be tweaked to adjust balance: enemy HP, enemy damage, enemy to-hit, enemy proficiency bonus, type of enemy, placement of enemies on high ground, items enemies have available, enemy AI, split exp between party members, the list goes on. I find it hard to imagine that it's impossible for the difficulty using a 6-person-party to be effectively the same. DMs do it commonly in tabletop after all, tweaking encounters when a player join or can't make the session.
It's reasonable to argue that Larian won't be able to implement it properly, but that's different than "impossible" and is a different discussion.
I agree that any 6-person-party option should be a hard toggle.