Building a wizard around summons has some drawbacks: First, you can only use one summon at a time. Second, some enemies have the Destroy Summon skill and can immediately kill that summon. Third, Summons tend to have poor accuracy, no skills to speak of and low damage even if they do hit (Physical-damage ones are better, though).
However, once your INT is high enough, the cooldown on skills will be reduced so that the cooldown for summons is reduced to 1 turn, so you can summon the same guy each turn if they die, so there's no real need to get more than a couple.
I think the Fire Elemental summon is particularly weak. Check the requirement, the lower a level you have to be to use it, the weaker it will be in comparison.