You can leave the fort without having to fight. In the top floor are two paths to leave with sneak. One is a oneway ticket, the other is more dangerous but would also open a way back.
Leaving the fort will give a lot xp, but also turn all magisters hostile.
Regarding the fact, that you are imprisoned, it is kind of fitting the story, that it is pretty tough to get around at the start.
But the main problem is still there: xp gain is pretty odd, difficult fights often give less xp than easy ones.
Using the ledge to sneak in was always the most obvious way to enter, in my opinion and regarding to killed the crocs. o.o