Games need a focus, otherwise developers are spread too thin, that is just the way it is. It's not as if I defending this exploit.

So you are stopping your players from using Haste and/or Quicken Spell to cast 2 levelled spells in a turn too? Because I consider this to be a larger balance upset compared to PnP.