Tried it briefly, and yeah, definitely felt weaker than a pure physical damage party.
Perhaps one way to make it work, without making the game too easy, or constantly having to go through two armor types, would be to make your melee use a shield, and your mage use physical damage spells (so mostly necro and geo)...
I might try that myself...
