The short answer is that when you wear armour, you are using that armour, and it has a particular method for calculating your AC, and you are relying on that armour to protect you. When you aren't, you are relying on your scales, which let you get to an Ac number in a different way. Mechanically, it's a matter of balance and the strength of perks.
If you want to think of it in an in-universe way, you could think of it in a way that's not too dissimilar to an unarmoured defence perk - the scales don't cover your *entire body*, and when you're relying on them to protect you, that includes moving in particular ways during combat to bring your scaled parts to the blows, instead of letting those blows hit your squishier soft parts. When you're wearing armour, you're not doing that specifically any more, because you're relying on the armour instead, so you don't gain that benefit any more.
