Its possible. Sheathing and drawing are both free actions. I know the warcaster feat lets sword and shield eldritch knights cast freely too as well as other things. I don't see a problem with scrolls. Frequency of scrolls is a different issue though.
The problem with scrolls is that any class can use them in BG#
A spell scroll bears the words of a single spell, written in a mystical cipher. If the spell is on your class’s spell list, you can read the scroll and cast its spell without providing any material components. Otherwise, the scroll is unintelligible. Casting the spell by reading the scroll requires the spell’s normal casting time. Once the spell is cast, the words on the scroll fade, and it crumbles to dust. If the casting is interrupted, the scroll is not lost.https://www.dndbeyond.com/magic-items/spell-scrollThe only exception is a level 13 rogue using the "use magical device" skill.
