Tough question.
He doesn't like to be manipulated in general - his trauma extends to any manipulation of him, including sex and relationships. But his trauma is much more complex than that.
I think he understands and likes the lust, the physical attraction.
The important thing is that he chooses.
I think Astarion realizes enough that Lae won't protect him, at all.
Lae was a distraction and fun. Maybe he wasn't with gith, he wanted exotic?
Here's an interesting devnote from Act 3.
In Tav's case it was lust judging by devnote, and manipulation, a seduction to get the leader of the group to choose his side and help him.
But when emotions come into play he doesn't know how to do it all, he's confused.