Instead of trying to get through an additional layers of armor, just let your mages attack the enemies that are weak to magic damage. Having at least 1 skill memorized that deals damage atypical damage for your class can be a great way to finish of enemies that have already taken heavy damage from that damage type.
Since the difficulty is supposed to increase as the game progresses, I would expect to see battles vs enemie groups with high, but unevenly distributed armor, which would be rather challanging for a party that primarily relies on one damage type.
