What if you're just bad at the game?
Mages can do so much more on the battlefield than any physical class. It's harder to quantify than hitting with a sword, but there's so many ways they can mess with things it just smacks of ignorance in your end.
You might want a point or two in scoundrel/huntsman/polymorph just to get some mobility options. You might want to get a shield. You might want some memory so you can remember more than 3 spells. You might want to remember staves have a 1 per round stats. You might want to remember elemental vulnerability exists and you can exploit them. You might want trio think about positioning and tactics.
The only problem I see is that earth is the only magic skill with different damage types and thus the most flexible. Ignoring the transient nature of summoning
