IMO doesn't make much sense to a magic spell to do damage to Physical Armor, maybe except for Necro and Geo. Also, your are suggesting basically to unify armor into one, but a lot more complicated.
The armor system was created to prevent you doing CC to multiple enemies easily like in the first game. It's not perfect (make CC lot less appealing) but it do good job.
