It comes from balancing. In D&D games spells are so powerfull that they have to tone down the wizard, otherwise it would be an impossibility to kill him/her.
Also it makes sense, that casting a spell in heavy armor is more difficult than in a mere robe.
