I'm not positive, but don't think so. I think the biggest things is that heavy armour lowers your movement rating, and magical/rare heavy armour is more likely to have bonuses to physical stats and skills and not magical ones.