My answer is that he is preoccupied with his fixation on the shadow curse and responsibilities as a leader. If you see things from his perspective, the grove almost completely fell apart and he later blames himself for not having the required leadership qualities. My interpretation is that he is a person who is very thoughtful and aware of things around him and he is processing what just happened and could have gone horribly wrong and realising that he needs to go back and fix the shadow curse to make things right.
After lifting the shadow curse he feels relieved and able to tend to his own needs and desires again and that's when he allows himself to get close with Tav and others again. Sure the whole brothel interaction kind of breaks this interpretation of him as thoughtful and aware, but I don't think that there is too much to read into the characters from that. It's cringy af and feels more like some teenagers One Direction fan fiction than some deep storytelling moment. His romance sex scene is honestly maybe the best in the game when it comes to communication and awareness. The way he seeks confirmation with words, eyes and body language and calmly leads Tav through it every step of the way is really well done. In contrast, his reaction at the brothel like some kid in a candy shop jumping in without any hesitation or tact feels like it's an entirely different person from the previous caring and sensitive character he has displayed.
Interpret things as you wish, but to me there is no conflict between his stoic rejection to letting Tav close to him in act 1 and the shift in the later acts as the storytelling about his fixation with the mistakes in his past is there to explain it.
