Another way to look at it is that a lot of magisters were sent out to the swamp and killed. There's what, 10-15 magister corpses and 10 or so magisters that you fight? Plus maybe several magisters died on the ship you came on. Not sure how many came on the second (the one at the end) ship, but that's perhaps 25 less troops than would normally be in the fort, if not more. IMO, the magisters probably deserve more permanent lodging, but it's actually a fairly realistic number of troops given their losses.
On an unrelated note, the final boss fight was a satisfying amount of challenge for classic mode, and it would definitely be very tough on tactician, though I might've been a level 7 vs. level 8s which made it more difficult. Hoping most of the fights are around that toughness in tactician mode.