Yes exactly! It sounds silly maybe, but I want to start relying on all three skill "variants": memorized, scrolls and potions (the latter is not a skill, but a buff, but it can act as a nice placeholder for some skills like Fortify). So on tough fights I'd have scrolls handy that act as cooldown indifferent backup skills.
Almost gamebreaking would be e.g. Death Wish. Scrolls also scale up to the caster's respective attributes (e.g.int) and abilities (e.g. geo). I need to test it a bit but I think even elemental affinity, as a talent, affects sroll usage.
Plus that would give a purpose to all that senseless hoarding.