I once added some bandits to the village (back when we had abandoned houses and empty cellars) and it completely felt wrong and out of place, illogical. I wouldn't just place combat for the sake of combat. It needs to make sense. But your feeling is noted.