I played 5 hours straight yesterday and got from the beach to the Shelter. Fort Joy's skeleton garrison of Magisters was annihilated by yours truly in around two and a half. Though I enjoyed killing all of those red-coats, in retrospect I realize that killing every Magister in a fortress at Level 3-4 shouldn't be that easy.
Let me break down what I mean by "easy". I'll demonstrate what I mean with an anecdote.
I was playing on Classic Mode - the supposed more difficult mode. I had just reached Level 3. After robbing the entire castle of its portraits and paintings, I decided I wanted to save the prisoner lady being judged by Orivand. So I snuck up right next to Orivand and his entourage, then sniped one of his mages with Sebille. Combat ensues. A swordsman Phoenixdives into my party of four, the mages unleash elemental abuse the like of which has not been seen since a baby threw a hissy fit, and Orivand contents himself with buffing his allies with movement speed and walking as close to my warrior as he can without actually getting into combat range. Realize that at this point, nobody in my party had actually suffered any damage at all. I proceed to Hammertime and Freeze all of the Magisters, kick them in the kidneys until they die and then wait for the rest of the enemies to show up. It was literally over in one turn - and they were one level higher than I was.
I hope that shows you what's wrong. I think the Magisters should be more powerful or have more CC abilities, because as it stands, it's just too easy for a determined low-level player to simply sneak into the castle and slaughter everybody from within.
I think one way to fix this would be to add more, more, more Magisters. The fortress feels very hollow - there are, what, like 50 Magisters in the entire place? Bump that number (whatever it ends up being) by 25% or 50% and make the place feel more alive! I really think this would bump up the difficulty AND bump up the atmosphere of the place.
Alternatively, perhaps there also should be more high-level Magisters present, or Orivand ought to be buffed. High Judge Orivand is a pathetically weak fighter - though admittedly I abused Rage and Flesh Sacrifice and Warlord a lot during these encounters. Perhaps Rage should be given more downsides, such as the massive elemental debuffs suggested in another thread. The point wouldn't be to actually nerf the good sides of Rage, but to add in a suitable downside as well. I'm not a game developer, though - these are observations followed by solutions I personally find appealing.
Also, the Silent Monks don't even fight me - I can just walk by them! Maybe that's as intended, though.
Once I also I had a bug where a Silent Monk talked to my character, asking about a murder (I had just sniped a Magister near Han's boat) - that ought to be fixed. They can't talk, right?