I've had the same bug, but I turned him down instead. Possibly the only solution is to not talk to him at all during the party? I don't know I haven't heard of anyone else having this issue. He still does do narrative-related dialog, it's just that I can't talk to him outside of things he's said he wants to talk about when we're outside of camp and encounter an event relevant to him.
