I don't find this problem annoying a many others I actually find this a strong point in their game (Bring the Hate). You see I find it better since it makes you have to rethink your whole strategy and go with something different. The only problem I find here is that if you want to do a All Wizard build then build you have no way on get the enemy to have no physical armor in order for you to cripple them.
Maybe if you want you can try a more mixed build between magical and physical dps, or you can even go lone wolf and have a wizard and warrior, rouge game. That's what I'm doing . I'm a Wizard and my friends a rogue and we destroy everything. So maybe try that and see how it goes
