I agree with Tuco. I want the conversation go to my pc. I hate it, when suddendly companions do the talking, especially the once, that are not good in it.
I had it exactly the same ...
Until last game, where i created 3 Custom Characters ... Barbarian (Host), Druid and Bard (Join) ...
Sometimes when i start conversation (as Bard or Druid), it switch me to Barbarian.

It sucks ...
I think Larian should rather allow us to switch talker at any time in conversation, would be thousand times more usefull and it would not really matter wich member of the party started talking, if anyone would be able to respond.