If Tav/Durge are so afraid and feel so bad as they show in the animations why would they even stay with him? Nothing is preventing them from just leaving him or even kicking him out. He can't use any compulsion because of the tadpoles.
Even after removing the tadpoles and without a prior breakup he's fine with Tav going with Karlach to Avernus. Such a monster. They also adjusted his dialogue for that scenario in the new patch.
It's weird he only makes Tav stay in the morning after the battle scene. Or maybe not. Maybe he realised you had so many chances to leave that you didn't take that now you're just messing with him and he doesn't take you seriously anymore. His story has become a confusing mess.
They also adjusted his dialogue for that scenario in the new patch. 
