Well, reload a save which none of my character interact with the portal yet. Grouping them together and use the main 1st character to talk to it seem to fix the problem.