I could not have said it better. I agree 4-man party should be where it is at.
So don't play in a 6 man party. What he's advocating for is the ability to do so. I think you're both shooting down the idea in spite of the fact that the game will be balanced around 4 people anyway so your experience wont change