That's right. Your hands and legs might be coated in steel, but it's no good if your barely clothed torso is hit. The overall Armour score is your average armour score for each hit against you. Since helmets and gloves cover less of you, they affect this score less.