Shields, the shield specialist skill and Defence rating (affected by Dexterity) factor into it, I believe.
Horrorscope: Try playing a game without magic. It's a lot of fun and gets rid of that whole "destroy everything effortlessly" thing. At least until you leave the Cyseal area.