Its possible. Sheathing and drawing are both free actions. I know the warcaster feat lets sword and shield eldritch knights cast freely too as well as other things. I don't see a problem with scrolls. Frequency of scrolls is a different issue though.
