That's true! I can see a situation where you're facing a clump of enemies with high physical armor and low magic armor. Your mage whittles down the magic armor, so you have your melee guy switch to a staff, then charge in and deal some damage instead of bashing his head against their physical armor.
The only problem I see with that is the skill, if I'm right, is entirely based upon weapon damage. So, for example, if you've invested in strength, the intelligence based staff won't be as effective.
Having tested it... While it is true people who don't invest the attributes into skills won't do as much damage...
It is still significant oddly enough. At least in the first eight levels.
But it is better for the warrior to instead dish out magic skills and/or Chloroform (an AMAZING magic armor destroyer) anyway.