Those goblins outside are already hostile to you.
Your only two options for avoiding that fight that I’m aware of are:
A) go deeper into the temple into the Underdark.
B) send out an invisible or super sneaky character alone. Send them up to the top of the temple and break down the cracked wall from the outside. They will need a hammer or some sort of spell like eldritch blast that does force damage. Then the rest of the party can sneak out through the roof.
I didn’t have any problems with the hags servants. You can sneak by that fight, though. I wanted to see if I could free them by killing the hag first. Only one servant is between you and the direction you need to go, so I used the minor illusion cantrip to send him away, entered turn based mode and had my party sneak down and jump across the waterfall.
