Passing via the fireslug way is far longer, the easiest way is:
Teleporting on the ledge with the door, letting stairs down climbing up, entering the chamber with the key or lockpicking the door, sneaking past the court room guys.
You can get a lot exp just with exploring afterwards or fighting the voidwoken wave attack. They should not be to hard.
The Voidwoken wave is not the problem. They're pretty weak. It's the rest of the enemies around. You need some decent gear and skills to take on the skeletons or else you'll get wrecked even if your level matches theirs. (And I know, I just tried that encounter.)
So it seems like you should clear out all the magisters from the fort before leaving it.
