Well it may be rather hard in the beginning depending on your setup, but later on between AoE CC and summons and charms enemies wont even have a chance of as much as look at your party
I almost never use my rogue's charm and the only summons I have are the two witch ones. And I'm still breezing through most fights.
Maybe I'm overgeared? I ran into quite a few uniques and legendaries.
