I found Oliver before Halsin was in my party. I played hide and seek with him and won, then did it again and his "family" attacked me. Now that I have Halsin in my party, I can see Oliver but he won't talk to me.