The area with the sleeping contingent and nearby wall is perfect to kill all the outside goblins. Oddly enough that won’t aggro the inside.

Inside you can get on rafters and kill everything in relative safety, especially if you steal all explosive barrels in the store room.