The issue, like with party controls, is that they don't seem to particularly care about it. Or more specifically, they seem to think that it's perfectly fine as it is and we are complaining about negligible cases, I guess.
That can't be the case. I have seen tons of feedback from players who want a larger party on discord, here on the forums and on reddit. I wouldn't call that negligible cases, but you never know. It would be far better if Larian came with some actual communication.