Enemies have four types right now
1) No Armor or Magic Armor
2) Balanced Armor to Magic Armor
3) High Armor, low Magic Armor
4) Low Armor, High Magic Armor
A good mage is effective against 1, 2, and 3.
None of your physical team mates is good against 3 (and some won't be good against the first)
No really build that memory. You will go from "Mages are weak" to probably being the second or most powerful member of your team. I am not just saying that.