Step #1:
Merge the Physical and Magic Armor into ONE Armor resistance
This allows Party synergy with multiple specializations (i.ex. Magic and Physical damage).
Pretty straight-forward.
Step #2:
Use RPG and Immersion to re-balance it
So that the weapon type/ element affect the Armor/ Health in an immersive way.
For Physical damage:
Some examples: Maces deal more damage to Armor than a Sword, but Sword cause bleeding once the Armor is teared down. Axes deal even damage (they can be used as hammers). Spears have high crit chance. Sneak Attack ignores Armor (a classic). Crossbow is better against slow Armored targets etc.
For Magic damage:
Magic schools affect Armor and Health in realistic ways, for example: Aero affect Armor more than Idro, but an electrified pool is even more effective. Fire can melt armor, so is more effective than Geo, but Geo is deadly once the Armor value is nullified, again, a pool of burning oil is even more effective regardless of Armor value. Poison ignores armor, etc etc
Of course, this is just my humble opinion, but I am firmly convinced that this is the way to keep things "interesting" while rewarding the player for experimenting and opening the experience to further Role Playing.
Take it or leave it, these are my 2 cents