Since there is no way to actually craft scrolls and there is no way to control access to scrolls I really dont get how its much of a big deal that Larian decided everyone gets to cast spells from scrolls.

For any spell, only spellcasters who actually get that spell can actually reliably cast that spell.