I tried to solve this with attacking them for the first time this morning and had a hard time too, having to restart the fight several times.
It wasn't until I did not talk to them and instead just started the fight by throwing spikes and fire at them that I won.
But it feels wrong to me to skip some story/characters to have an advantage in combat.
I've had other occasions where actually triggering the story and talking to the NPCs put you and your party in a really bad position for the start of the fight. All of your characters are clustered and the enemy starts off with throwing fire or acid at your entire party.
Splitting the party in case a fight could emerge sounds very tidious.
One trick that helped me in this fight that came to me very late was just destroying the drum with an attack, to keep them from summoning reinforcements.
