The main thing I want changed is that spells should have requirements beyond 1 point. This is a pretty common opinion though and I don't feel strongly about about the bonuses one way or the other so instead I'll posit another thought.
What if they added requirements on gear to be worn? Respective defenses for armor and respective hand requirements for weapons. That would give us more incentive to invest points in defense beyond 1 point. If they did this they would have to buff the return on investment though it it would be a harsh tax for the "tankier" member of the group.
