Why don't you exercise self-moderation and just use the core keybindings?

Haha, jokes aside, same problem happening to me. Adding to that, sometimes even spells that I removed from the bar comes back...