That's an issue in m&K as well. There is no way for the game to remember or not assign spells:
1) change weapons: things get messed up
2) remove add new spells: things get messed up
3) equip/unequip items: things get messed up
4) shapeshift into an animal and back - every bloody spell I removed previously gets shoved back into the hotbar.
From little I played with the controller, I imagine the problem is even greater there, as radial hotbar is the only thing there, and removing/assigning skills is slower and more clunkier.
