The problem with this is: cleric spells come from their deities and wizard spells come from the Weave. They are different types of power, at least in the D&D's Lore.

I like the idea that everyone can use scrolls, with different degrees of success. In Pillars of Eternity it is necessary to have Lore skill points, in Pathfinder Kingmaker you must have Use Magic Device skills, etc.