There are a couple of issues that I see with the armor system:
It's just an additional HP bar impacted by either physical or magical dmg. The name 'armor' would imply (for me at least) some sort of resistance mechanic that negates a portion of that damage.
It's against one of the classic RPG tropes of having a mixed party; why would you do that if splitting the damage type among the party is actually detrimental? This is reinforced by how the CC works now - needing to first remove that armor type means again that you shouldn't split the damage...
Anyway, I don't think an easy solution exists. I tried working with what is in the game and tweak it but I didn't like what I got. But what would I do anyway is change the itemization system in not allowing both types of armor on the same items - you should choose one to specialize in. If you want to wear both warrior armor (with physical resistance) and mage-relevant armor (with magical resistance) you can but you will be much weaker because of that. Versatility should have a big drawback.
The biggest issue will remain the bosses as they seem to have magical and physical armor of equal values. And for that I can't see an easy way out.
