Heavy armour could also just cause a longer casting delay due to the reduced range of motion for hand gestures.
Siege of Avalon had a stealth penalty for stronger armour, to account to the noise chain or plate mail (etc.) would make, so enemies could detect you at a greater range. Not that it did them any good. <img src="/ubbthreads/images/graemlins/badsmile2.gif" alt="" />