I disagree because there may be reasons you swap out to another character. For example if I'm talking to a wizard character I might want Gale to talk to them because his magic knowledge may be able to influence what happens.
There's also the awesomeness of having your party members talk to eachother, without your main character.
It's a feature you don't see in many games, most probably because it means a lot more voice acting. That said I'd be okay with say the main character butting into conversations sometimes to make it so fewer voice acted lines are needed. Seems like a fair compromise.
