I think one way to fix this would be to add more, more, more Magisters. The fortress feels very hollow - there are, what, like 50 Magisters in the entire place? Bump that number (whatever it ends up being) by 25% or 50% and make the place feel more alive! I really think this would bump up the difficulty AND bump up the atmosphere of the place.
I'm not at all against the idea of more magisters. However, if you add more, it just adds a little more absurdity to the living conditions of the magisters:
It's absurd that the island has been used as a concentration camp for YEARS, and yet there's nowhere for the troops to sleep and nowhere for them to prepare a meal. There's no armory or storage area to keep their supplies organized.
I think
Fort Joy needs another floor, which contains a barracks and kitchen for the Magisters, and some other things as well.
Maybe that could be placed between the prison and surface floors, so taking the stairs up from the prison puts you in the barracks, and taking the stairs down, or the door in from the surface level puts you in the barracks.
