I think he is more comfortable in the astral because he has more confidence in his abilities there, but he expresses desire for physical things throughout the game. The thing about liking the PC's musk and the whole conversation about how stimulating fighting together is being the most overtly sexual. It's something he does like to give a try again when with someone he trusts - but not when it becomes a (perceived) competition like in the brothel. For different reasons the brothel is too much for both him and Astarion.
