Removing the ability to use scrolls from non-casters would go a long way to balance them.
I don't disagree ,but sometime I get the impression that some of you are missing the point:
This is not an "oversight in balance" from Larian, this is exactly what they want... To throw to the player so many tools to be able to compensate even for the most ill-advised party composition.
They WANT everyone being able to use mage spells even without having a mage.
They WANT to trivialize the access to "special abilities" like Speak with the Dead and Speak With Animals.
They WANT unlimited and super-cheap respec to be a far-too-convenient tool bordering into the potential cheat.
And so on.
It's "messed up by design", because they are too afraid to ask the player to COMMIT to any long-term decision and its consequences.