I still view Fort Joy as nothing more than a "processing centre", a fairly transient concentration camp. It doesn't need too many guards, nobody's really there long enough to organise themselves into a serious opposition and the factions that do exist amongst the captives don't get along anyway. It only needs a skeleton staff which seems to consist of assorted zealots and ideologues, and probably a few who were sent there as punishment, obviously in addition to the senior magisters who are running the show. And as such, a few tents will suffice: why give the dregs of the Holy Order comfortable accommodation? The maniacs will enjoy the hardship and the reprobates don't deserve to.
Anyway, that viewpoint works well enough for me.