Do you have the correct scroll to activate that particular portal stone? If not, and I'm not sure about this just check, maybe one of the people in the area have it. Other than that, all I can say is you'll have load to an earlier save file.
Don't make that harsh decision though. <img src="/ubbthreads/images/graemlins/exclamation.gif" alt="" /> I'm not going to lie and say I'm a pro at this game, I'm just a 'not-bad/pretty-good' gamer. Wait for one of the better forum answerers (like Barta, Umbreil, Raze, etc.) to tell you whats what first.
Just check if you have the scroll first.
Killerzzz