But your own characters also tend to have skewed armors. Items with high physical armor tend to have more strength requirement, items with more magical armor have more intelligence requirement. Enemy fighters that usually have more physical armor are not more of a threat to your own fighters than enemy mages, since fighters mostly deal physical damage.
Most enemy groups in the game are relatively balanced. Against a balanced enemy group, a pure physical/magical party can take down the more vulerable half of the enemies much faster. In a strategy game less units = less damage per turn. Then the party can handle the remaining half more safely.