You can hinder physical with evasion. The problem is that its barely used in the game and the spell only lasts 1 round. There is at least 1 NPC in the game that has a permanent evasion aura which makes her hard to hit (50/50).

I did not have a problem finishing the game with magic but you are right, it is far slower than physical. If I have time I want to make a overhaul to balance things out better but I simply dont have time right now. I need to prep assignment and tests for next quarters programming class and then finish up a solar install for my parents before working on mods lol.