I'd like to see something closer to this:
1. All attacks deal some amount of damage to all armor types.
2. You need to eat through both Armor types to start damaging health with any kind of attack
3. Physical attacks deal 75% damage against Physical Armor and normal damage to Magic Armor
4. Magic attacks deal 75% damage against Magic Armor and normal damage against Physical Armor
5. Maybe those could be swapped if that makes more sense, with Physical attacks being the counter to Magical armor and vice versa
6. Some attacks would deal bonus damage to Armor, something like "Sunder Armor, deals X - Y damage, and a bonus A - B damage against Physical armor"
7. These special attacks could come in every permutation, e.g. a physical attack that deals extra damage to physical, a physical attack that deals extra damage to magic, a magic attack that deals extra damage to physical, and a magic attack that deals extra damage to magic
8. The Resist system can stay the same. This way you'd try to focus down an enemy's weakest armor type and then use a crowd control spell of that type before going to work on the other armor type/health. Right now, by the time you get around to being able to CC someone, you may as well just kill them instead.
I think this also solves the issue of hybrid characters and mixed groups being less optimal. If you routinely need to eat through both physical and magical armor, then a character/group that can do both isn't at a disadvantage.
You also don't remove the option of pure physical or pure magical groups because of those abilities that damage the opposite armor type. You even have the perfect system in place for those abilities. Take that "Sunder Armor" example ability, combine it with any elemental skill book, and get Sunder Magical Armor. Now your pure physical team isn't weak to magic armor anymore.
This is too much for an official change, but I assume something like this could be modded.