But what would I do anyway is change the itemization system in not allowing both types of armor on the same items - you should choose one to specialize in. If you want to wear both warrior armor (with physical resistance) and mage-relevant armor (with magical resistance) you can but you will be much weaker because of that. Versatility should have a big drawback.
Have you actually seen mid-game armor types? Because that's kinda how it already is.
Strength-based armors give a LOT of Physical Armor and a tiny amount of Magical Armor. Intelligence-based armors give a LOT of Magical armor and a tiny amount of Physical Armor. Finesse-based armors give a moderate amount of Physical armor and a lesser amount of Magical armor.
You don't get both unless you split your attribute points between STR and INT, which has a drawback of wasting attribute points.