I honestly wondered if I had done something wrong in Phantom Forest, simply because there were so many named npcs that aggro'd instantly.
For the record, I think that 'large scale' fights are the worst, in this game. In particular, the battle royale at hunter's edge made me want to self-terminate. The combat works best when there's less than 8 enemies, in my opinion.
The problem with high level combat isn't the lack of enemies, it's the lack of danger. In my experience, it didn't particularly matter how many enemies there were. Unless they outnumbered me 10 to 1, there was no way for them to actually put me in danger. I don't think I had any of my people drop below half health, even during the final boss fight. On Hard, of course.