The Harpies are quite annoying. Mostly because bad rolls can completely ruin you.
Nere fight can be annoying too since again, there's more chance to get ruined by an unlucky charm (Now also with the potential to get punted into lava)
The multi-Mindflayer fight if you opt to save Zevlor. More charms are more fun (NOT).
The Death Shepard duo in Mountain Pass. Their infinite resurrections make it just super annoying because you have to kill both Shepards at the same time as a result, while all the ghouls (Including the one that's farting out the Stinking Cloud) get free reign because again, you can't take them out because free action resurrections just bring them right back...
I stumbled upon the Death Shepard herd on a road bifurcation, and this became the perfect choke point to hold them at bay, with Wyll's Hunger of Hadar spell. I only had to take care of the few that managed to slip around, while firing missiles and long range spells to the ones inside. When one did manage to move out, my monks "push flurry" or Lae'zels push manoevre or Wyll's repelling blast pushed them right back in, after they got a radiant shock from Shart's guardian spirits.
