Fair enough, here's my idea, which I'm sure has some holes in it. I'm also sure that it's unlikely to end up in the game.
Keep both Magical and Physical Armors, keep them blocking Physical and Magical status effects...
But split the damage so that Physical attacks do, 67-75% of their damage to Physical Armor, and 33-25% of their damage to Magical Armor. Magical attacks are the reverse (67-75% to Magical, 33-25% to Physical). This allows for better synergy between mixed teams. For the moment, let's say that the damage split is 75-25, that seems to make the most sense, I don't want to weaken the damage types versus their type of armor too much.
If one type of armor is gone but the other is still up, damage from the opposing type can seep through to Vitality at the same rate.
Let's take a enemy, call it "Vazgar" which has 500 health, 0 Physical armor, and 300 Magical Armor.
So a Magic attack which does 100 damage and inflicts a CC, hitting Vazgar. The spell will do 75 damage to the Magical armor, and 25 damage to health. The CC is blocked because magic armor remains. Vazgar now has 475 HP and 225 Magical armor.
A Physical attack does 100 damage and inflicts Bleed hitting Vazgar will do 75 damage to vitality and inflict Bleed (no Physical Armor to block it), and 25 damage to Magic armor, reducing it to 200.
Obviously once both armors are gone all damage goes to health and statuses will land. The amount of health provided to all characters might need to be increased slightly to compensate for the faster rate of death thanks to armor no longer absorbing all the damage.
This is mostly the current system, but it has a dramatic effect on the effectivenes of mixed parties, because now attackers of one damage type help out attackers of the other damage type a little by reducing the opposing armor, and once one type of armor is depleted, some damage can seep through directly to health.