For me, it's obvious that a relationship with him is more sexual than based on love. In the game, there is no way to say "I love you" to him, nor does he confess this feeling to Tav. I assume that it is a physical fascination and a relationship based on physical closeness. Halsin does not want to call it a romantic relationship.
As for the scene in the brothel, it is a specific place that triggers less subtle strings in people. I'm not defending Halsin's strange responses, but Gale could have let go of his presence there as a projection. It was immature, embarrassing, and not even funny.
