One of the things that helps a 3 physical/1 magical or 1 physical/3 magic party is multiple enemies with a big armour pool of one type, but low or none in the other. This is a part of encounter design which can be used to balance this to an extent.

The way you want to solve this actually means you need a party of all damage types and effectively can't have a full magical or physical party, as all CC is blocked by the other damage type. It's not solving the problem, it's putting it in another place. You would also need to actually merge Physical/magical Armour entirely, at which points it would solve the issue of mixed groups.

Another idea would be to keep the armour types, but have magical armour get damage reduction to magic damage, and physical armour reduction to physical damage, but unlike now, when the magic armour is all gone, the damage goes to physical armour instead of vitality, the magic damage would deal full damage to the physical armour and not be reduced like it was against the magical armour.

Having a huge number of variables like with the 100-0% armour absorption with anything not absorbed going to vitality makes it a lot harder to see at a glance how much damage you will do to both vitality and armour, slowing down the game. I don't think you need a variable system like that, and can achieve the same effect much more smoothly by always splitting it 50/50 armour/vitality.

So taking some of your ideas and merging it with my own, spitballing some numbers just for clarity.

1. Physical Armour gives 50% damage reduction to physical attacks, Magical Armour gives 50% damage reduction to magical attacks. They absorb 50% of the reduced damage, the remaining 50% going to vitality. When one armour type is destroyed completely, damage then goes to the other armour type, but will not be reduced and so do more damage.
2. Debuffs(CC among them) have different thresholds, like 0% armour for stronger debuffs or 50% armour for smaller ones. Armour is combined for calculating whether the threshold is reached, with the armour type that's being targeted (magic for Dominate Mind for example) counting double. So 2*Targeted Armour(+Other Armour). So someone with 100/200 Magical armour and 100/100 physical armour has 300/500, or a 40% chance to be affected by the Dominate Mind, or 80% chance to be affected by a smaller debuff like Burning. (50% threshold being reached counting as 0% chance to resist, with 20% chance to resist at 60%, 10% at 55% etc.)
The same Dominate Mind against someone with 200/200 Magical armour and 0/100 physical armour would have a (2*200/200(+0/100) 20% chance to go through, as he'd have 400/500. This way, magical armour protects against magical attacks and debuffs, and physical against physical attacks and debuffs without being unrelated.

Now, this was mostly done because I thought it was an interesting idea to combine it all, I do think it's fairly complicated but not overly so. Numbers are obviously rough just to give an idea. But I think a 40%/80% threshold for big/small effects would work best like this, with vitality being reduced relative to the armour people have in the game right now, to make vitality meaningful and have the chance to die from vitality damage before armour is gone.

Curious what people think about this, at least.