The game is very easy in general, and I didn't try and make particularly amazing builds or anything. This isn't the true hard mode though, so I'm not surprised, but even so, classic mode almost feels like how Explorer mode should feel to me. Not sure if it's the enemy stats are too low, or just the mechanics make the game easy.
Regarding Fort Joy itself, I do think there could be a couple more guards in the actual ghetto, and a second story for in the interior like Stabbey suggests. What if the magisters had some kind of (melee?) ability to send a current through your collar and stun you? Obviously this would make magisters super dangerous and scary unless you removed your collar, though maybe you could modify your collar to disable that stun effect but keep the source muting.