I feel that, especially since patch 6 announced a platonic path for him, some interactions are not setting / reading flags correctly and are, for this reason, bugged. The one I mentioned is an example, it clearly doesn't check if you have had the conversation about his previous lovers. The comments about the post party conversations seem to fall into the same category.
The brothel and the banter though, seem to me to be disconnected from his usual romance path. I currently can't check it, but doesn't he also invite himself into the brothel scene if your approval is too low for his usual proposition? The comment about swimming with Shadowheart is also simply triggered by being in a relationship with Shadow and having had her act 3 scene, the goldfish one is Shadowheart's comment if you are in a romance with Halsin (no matter if she is involved or not).