You started a fight with them, by ignoring their checkpoint at the entrance to the camp.
Those NPCs, with whom the battle was started, memorize this and became hostile to your party.
Hostility is the result of your own actions. Why? All summoned beings are members of your party, aka the party is responsible for their actions. If your dog, kitty, parrot or King of Wind Demons Pazuzu bit anyone, you are responsible for it.
