There's a ladder right by the waypoint of the goblin camp that leads up to a ridge with plenty of room to move around. Climb to the top, destroy the ladder once you're up there so the goblins can't get up there and then rain hell and damnation down on them. That's what I do. Works everytime. Like shooting fish in a barrel.
Once you've killed everything you can just fast travel to the same waypoint down below to collect your goodies, since you destroyed the ladder.
I initially used the alternate entrance - the one you access by climbing the vines by the "Owlbear run" area. You go up there, kill the drunk/sleeping bugbears, and then have someone who has a bludgeoning weapon smash through the wall (Gale used his quarterstaff...LOL). Then you can head back out that way.
When you exit, you're fine, even though everyone is hostile. Only around half of them have ranged weapons, and they usually can't hit you because you have elevation on them. The ogre might try and throw a single boulder. Take out the archers/spellcasters first, since they are the only ones who can even hit you. Once they're gone, the melee goblins are totally helpless.
