You can leave the fort and still be able to interact with the magisters (sort of).
The magisters only become hostile towards the specific inmates who leave. So what I did was have my thief/recon dude leave since he sneaks best and he's telekinetic. My other three characters remained inside Fort Joy. The Magisters became hostile towards the escaped inmate, but not the other three (in my case - Fane, Ifan, and The Reddy Prinze Jr).
Anytime you want to TP your thief/recon back into Fort Joy, have one guy talk to Magister Alise (the archer on the LEFT side of the main gate) so she turns away. Have another one talk to inmate Dominick (the dude praying at the statue of the Divine). With these two looking away, a hidden party member can TP into the Fort Joy Square point without being revealed.
You can also do the reverse, sending out your 3 hardest hitters and keeping your best talker inside the Fort.
Specifically, the points I found at which hostility commences were;
- Any inmate who descends the partial ladder outside of the courtroom
- Any inmate who utilizes the sewer pipe in the upper cellblock
- Any inmate who allows him/herself to be transported by Han
- Any inmate who crosses the drawbridge (I'm not sure specifically where the trigger is on the bridge)
- Any inmate who teleports to any point NOT beginning with 'Fort Joy - '
- Any inmate caught inside the courtroom (duh)
- Any inmate caught in the area beyond the main gate guarded by Magister Borris (also duh)
That's all I have at the moment. Another thing that might work (haven't tried yet) is to dismiss a 'burned' party member and recruit one who's still inside Fort Joy proper. I don't know where the burned inmate would end up, or if the magisters would remain hostile toward them.