Problem is, scrolls serve two purposes. They makes wizard's spellbook, and they make up for not having a divine or arcane caster. Just like how everyone can try to pick locks.
The difference is: 5e allows for anyone to try and pick a lock as long as they have the tools to do it (though their chances may be slim without having the proficiency). Scrolls, however, are only meant for casters and can't be read by just anyone.
Removing the ability to use scrolls from non-casters would go a long way to balance them.
As would using D&D 5e rules for certain other abilities (Cunning Action, Haste, etc) to avoid multiclass cheese...
And of course removing the damned elemental damage from most magic weapons.
