Duergar freaking out about spiders and skeletons, people in the druid camp freaking out about some of the animal companions or wildshape forms, or the goblins panicking and telling you to go away because you have something summoned when half of them have or summon wargs doesn't make any sense. And nobody should be freaking out about minions in a combat area. A sweep of which people are bothered by which creatures should be done or, if you don't feel like doing that, at least remove the behavior for some of the areas with NPCs, such as the
goblin camp, Duergar dig, and Moonrise Towers
.