I agree OP, as someone who does fencing IRL heavy armour should be reducing all damage to the point where you should not be taking any damage. Heavy armour can even absorb the full swing of a halberd, mace and long sword. Only axes / bear form and explosive magics should be able to damage heavy armour.
The drawback to heavy armour should be that your forced to repair it after every fight and that it should be expensive to do so, that way the player must decide if they want to accumulate gold or being protected.