I don't think that you should be able to 'have your cake and eat it'. By not bringing an arcane or divine caster - you simply shouldn't get access to spells. And in any event in 5e there are so many ways to get someone who can use scrolls - be it a bard/eldritch knight/arcane trickster etc. If anything - it would actually make subclasses like eldritch knight and arcane trickster a lot more useful.
As I say - I doubt Larian would do this as restricting players is counter to their playstyle - but its absolutely the correct thing to do here in my opinion.
