I like this idea. The only possible issue I see is that it means your cooldowns will be stuck indefinitely if you are not in combat. I can see that possibly causing problems, like if you want to heal (or revive using your resurrect spell) but you can't because you're not in combat. I suppose that's not that big a deal, though.
Actually it is, considering that you can use spells out of combat as a tool for solving quests. I crushed a door, for instance, using my spells.
