I think it would be okay with three small changes:
1. a hidden window (like the combat screen) that contains all the character's potions,
2. another hidden window that contains all the character's scrolls, and
3. a command to bring up a 'spellbook' overlay where spells can be selected.
Other than that, keep it the same. Frequently used stuff can still be on the hotbar, but the scrolls and potions have their own drop side window to keep the hotbar from getting cluttered. And while the frequently used spells can be on the hotbar, the rest of the spells can be brought up by pressing something like "S" on the keyboard.
