That's why you don't respec - get enough in all elements to cast the skills you want (with gear) and the rest into boosting your crits and polymorph 5 for skin graft and apotheosis. I find wand/staff builds ok early/mid game but lategame there's no point since you do way more damage with your source skills (and they usually oneshot enemy magic armor if you choose the right one). The stuff that makes wand and staff builds good, like venom coating and surface damage scales pretty poorly into the lategame as well.
