I don't remember this being an especially hard fight in my first playthrough with no foreknowledge. The party clustered inside the tea house door or just out front, and we picked them off as they arrived, with maybe a melee fight vs. 2 at the end. I didn't use any surface effects or other cheese.

I can see where aggro'ing them in a group outside, close to the party could be tough. I'm just at this point with a playthrough on this new version (taking my time), so I'll see if it goes this well again. Maybe they increased the difficulty?