Regarding the failed CHA check triggering combat, I agree that's not great design, however, it's pretty clear going in that you're not to faff about and anger the goblins, as it can go poorly for you.
Both of those combat encounters are very doable at-level using the right tactics. For the priestess fight, she doesn't actually aggro every goblin in the area. It's a wide area that gets pulled in, but if you keep your party in her little room and funnel them in you can limit the number of combatants and it becomes more manageable. It seems to be the war drums that toggle the entire area to hostile, and there aren't any near her for that fight.
For the fight outside, I recommend doing it after you've cleared more of the camp and they're all hostile. Fast travel to the goblin camp waypoint (this starts the fight) and get your party on top of the wall where you spawn. From here you have advantage over the entire field, bad guys trickle in one after the other and are easily 1-2 shot. Fire surfaces go a long way to limiting how fast goblins come, as the AI would rather risk arrows than damage from surfaces. The ogre's pathfinding sucks and he can't get to you. It's an easy, if somewhat tedious fight if you do this.