Going to the brothel is tied to the main story. You need to talk to Voss and Raphael in there. You might just be walking around and talking to people to hear gossip, or whatever.

It's also part of a side quest. The lady that runs the brothel thanks you for finding the corpse of one of her employees and says the reward is something the drows have. You can go to them naively thinking it's not just about sex and happen to have Halsin with you because he was acting normal up until then and you liked him as a friend and found him useful.

You're not inviting him to the orgy unless you specifically go there afterwards with the intention to initiate that scene.