Add to this the notion that the casters most inclined to use concentration spells in the first place are Pact of Blade Warlocks, and it seems at least a little more annoying I suppose.
Their whole thing is being a melee combatant while having some suped up Warlock concentration spell active (or just hex), which means you aren't doing any of the things others casters might do to avoid being hit.
