Having armor not be affected by actual damage, but only taking damage from effects is interesting, but given the cooldowns of CC spells, you'd have a hard time breaking armor with CC effects before the combat is over. You spend your load of CC spells to break the armor on one enemy, but then what? You no longer have CC spells to actually apply to the enemy. And that is for ONE enemy.
Only surfaces would really work for breaking armor. The net result would be that it would be easier to just wipe out enemy health instead of bothering with CC at all.
