The total armour listed in the stats pane is for just for your information. The armour class of where you get hit is used in the calculations to determine damage.
For how defense (and offense) is calculated based on agility, see
differences between classes (minor spoilers)
or
Early Starter Tips (NON-Spoiler Version) (contains much of the information in the above topic).
