That's the problem. I will have no problem if after the slave joke and you accept, the dialogue go slightly further where you actually get to express the desire to go with him and he express some kind of desire to have you around as well.
The current one with the "You go sleep with Sadha then, I will go on a new adventure." is a cuck move, one where you completely accept the defeat.
