You can only poison the booze before you have killed the three leaders. Once you have killed them - like OP did - all the goblins outside automatically aggro on you.
But yeah, it's much easier when you clear the outside first, since you can take them all on in groups. Poisoning the group in the main courtyard cuts down their numbers, and weakens the remaining. And as long as you keep your team to the "chicken chase" side, you won't aggro the group on the waypoint side. Till your ready for them. Position your ranged party members on high ground before starting each fight, and they all go pretty easily.
