I think higher difficulty should influence enemy resistances to impede magic as well but that's off topic.
Anyways at the current system, you finding it too low levelled may not be the case for someone else. E.g. If I went with charisma for all my encounters, I'd receive a lot less experience than having fought thru every fight. With experience being a finite resource, I could be a couple of levels lower than your play through when I approach the boss
