Controlling you to do what? He's not doing it for fun. It took massive risk and effort for him to communicate with us while he was fighting off both Orpheus' guards and the Brain's influence.
Controlling you to survive. Do you know what he does when you approach that bridge to the Upper City and it's Gate? Have you tried ignoring his pleas to turn back? He takes control of your body and teleports you back, his words being "If you do not listen to reason, I will solve this myself". And for some reason it's the only time (aside from the first tadpole extraction from Edowin if you fail skillchecks - Durge has no skill checks for that btw.) he does take control of Tav.
