Once I had to walk really close to the portal and past it a couple of times in order for him to appear. Probably a bug unless you waited a really long time to go to that area, if you do this then I think he will not appear.