I sorta see what you're getting at and I'm not saying it can't be fun or isn't good.
If you crunch the numbers though:
Seems to me that regardless of armor, by doubling down into a specialization for damage type you're better off giving yourself the tools to simply blow through the amor and then continuing on to blow through their health.
A heavily focused two handed warrior can basically kill pretty much all the regular enemies on one or two hits despite armor cause of how high you can stack everything.
In the same vain, a heavily focused mage with air/fire up high will blow through the magic armor and CC/burn/damage anything.
And once the defenses are down will be much more efficient in the actual killing and will certainly be better at liking things not armored for their damage type specialization.
There's also the fact that I'd give you're mage enough Memory, I doubt you'll be hurting for cool down spells all the time. And wands offer elemental combos from a distance for that odd turn you have nothing and one of your spells have normally set up for a combo.
What I get from you is that melee staff wizards are more of a dual class build that can focus on either physical or magical attacks depending on context......but we've always seen in these games how in combat heavy specialization can create overpowered builds that jack of all trades can't reach. Except in the case of using a lot spells for full field control.
