You can get to fort joy at level 1 if

1) You don't kill any of the magister
2) You don't do the death fog part
3) You don't fight the 2 voidwoken at the beach before you recruit all your companions.
4) Near where you find Fane, don't climb the vine up to the hiding camp

If you do this, you should barely scratch level 2 and you can return to kill the void woken for the missing exp.

EDit: Having tested this, if you do the Ifan quest, you get level 2 guaranteed. I tried to avoid everything and even climbed into the fort with a ladder but the Ifan recruitment part just push me up into the level 2 range with 200 extra exp.

Last edited by Ellezard; 15/09/17 10:53 AM.