If u met more enemy with high physical resistance, you won't think that pure physical or magical team is a good idea anymore.
If we talk about challenges to a specific party makeup, the toughest enemies in the game are bosses, they tend to have more balanced armors, and a balanced party would have more trouble handling them. So it's likely easier to get stuck because of game difficulty if your party is balanced.
There is rock paper scissors for skewed characters in the game:
pa and pd > ma and pd > ma and md > pa and md > pa and pd
pa/ma = physical armor/magical armor
pd/md = physical damage/magical damage
I feel it's not a good idea to have such counter relationships in an RPG game, it's difficult to design encounters and make them fun no matter what your party makeup is under such system.