This kind of sounds more, like it would make the issue even worse. What exactly happens when physical damge hits magical armor or the other way around?
My suggestions would be still, combine both kinds of armor. If you carry mainly heavy armor pieces your armor is more vulnerable to magical damage, if you carry mainly light armor the armor is more vulnerable to physical damage. If you carry mainly leather, none is more effective, but the armor amount in total is less high. Or the 'type' gets defined by the body part of the armor.
