I would certainly not mind if scrolls would be locked only for casters ...
Even tho i cant quite understand why, since if you dont like it all you need to do is litteraly anything else than casting from scroll as a non-caster ... and you should have exactly the experience you want.

You could take it a step further and make divine spells unlock with Religion (like you’re reading a prayer). Maybe Druid spells could use Nature.
This sounds like horrible idea ...
Yes im admiting it, im min-maxing and im not ashamed for it even a little bit ... but my Clerics, nor Druids never had and hardly ever will have high Intelligence score. :-/
If something simmilar should have ben implemented, then only (and strictly only) with alternative ability scores (Nature + Wisdom bonus).
You could even make the skill check to successfully cast a higher level spell based off this relevant skill.
This sounds a little too far ...
You would make skill check if you even can cast a spell ... and then you would do another roll for if that spell will even hit someone ... ? :-/