With the Zents, did you walk through the Illusion wall to the Lift? This caused hostility for me. Which was weird because I fought and wiped the first Zents out and the lower section were friendly until then (I delivered the cargo, having opened the chest and Zent lady kicked off). Can't comment on the Goblin Camp. did you fail a check to pass the guards inside? They kick off if you don't persuade them to let you in. I usually go in through the fragile wall at the top to avoid this.
