I did not experience this but something similar and with essentially the same encounter. I walked through the goblin camp and took out the leaders inside one by one. I was able to do this at my own pace until I took out the last one at which point the entire camp became hostile and when I exited the inner area it automatically turned into a massive encounter without any warning. You can take out the lesser bosses without triggering any further combat so it came as a surprise that I could not just walk out of the goblin camp as I had walked in. I ended up reloading a save and just warping out of there but its just that the last thing you are expecting after clearing a dungeon and finishing the boss fight is to walk back through an area that was no hostile and immediately get dumped into a 16 enemy encounter.
I was kind of chuffed with the idea that I had stealthed my way in and out using the goblin prisoner but it didnt really matter in the end because some variable got ticked and everything went hostile.
Are you sure none of the goblins banged a drum? I think the intent is that the drums toggle the whole camp hostile and I personally wasn't able to keep them away from them so I'm not totally sure.
