lol, i don't know, but that npc took out three of my characters immediately. i managed to win with the last standing character, but it was touch and go for a bit.
i put this in a spoiler tag, not because i have a definitive answer, but just to be overly careful about spoilers.
i think it has something to do with the amount of gold the characters are carrying and the existence of the minions all around, adding strength to the toll collector. i haven't spent a lot of time researching it (or replaying it) because i wanted my first playthrough to be completely organic, without any idea of what to do other than what i can figure out in the moment.