I think there's a subtle benefit to having damage for both types in the form of resistances. If an enemy has high resistance to magic, physical is obviously going to shine. Likewise, some enemies have negative resistances to certain elements, and will die way faster if you focus on hitting them with it. Also, some enemies have way less physical armor than they do magic armor, or less magic armor than they do physical, and having access to both damage types can help if you split up the fight.
Maybe a good solution would be to introduce more enemies that have strong resistances to one damage type over the other, to make it more difficult if you focus on one damage type. Like if you go all physical, and an enemy not only has high armor, but has fortify, a shield, and maybe even some physical resistance (if physical resistance became a thing), then having a magic damage character would be beneficial.
