OK, I have NO IDEA who to listen to. I found this great orange leather armor that adds intelligence, but its a leather armor, NOT a robe... It seems to be affect my movement speed or am I just paranoid?
Thread is bollocks.
Armor penalties exist - you'll note them in AP / Speed penalties, which are mitigated by the responding skills.
So, yes: your mage can wear heavy armor, but will have some AP / Speed penalties using it without the responding skill.
TL;DR
Mouse over the armors - you'll see the penalties are listed.