There's a reason you can't turn down the difficulty, and the reason is because if you turn it down, you didn't beat it on Tactician. I ran in to the same problem: shitty, non-optimal builds, ran out of XP, hard stuck in the dungeons. Time to head back to the drawing board.
Figure out what is getting you ruined in fights, and restart with a stronger build/composition. The entirety of Fort Joy can't take more than a few hours at most for someone who knows the area, which I would imagine you do at this point.
If there is an option to turn down the difficulty at a later date, it absolutely needs to be a one-way road. If you drop it down, you stay down.
Edit: autocorrect
Last edited by TheArtisanWolves; 16/09/17 08:13 PM.