Single element mages can do comparable raw damage to physical until you run into resistances. Then suddenly your damage is not so hot or you even heal enemies with it. Unless you're willing to respec for each encounter which I find completely masochistic.
Also, any given elemental school on its own has worse utility than warfare, which has abilities for every occasion. Mages have to use more than 1 element to be viable.
The best elemental lategame mage build I used was actually 5 in all elements and poly + a little bit in scoundrel (with gear of course) + glasscannon. Then you can spam max tier source spells under apotheosis + adrenaline (potentially recharging them with skin graft in the end), and they do way more damage than anything else mage has, even when compared to if you had 10+ in the relevant skill. You also usually have at least some spells that would hurt. Still this is weaker than a maxed physical build that also requires no skill to build or play.
