Considering how absurdly powerful spells become at spell level 3 and above, giving willy nilly disadvantage against spells will be game breaking.
Imagine being able to guarantee that power word kill hits just because you have the high ground, good bye any sort of challenge.

You can still impose disadvantage against saving throws, within the mechanics as is, most notably would be Hex.

Last edited by Mxiio; 14/10/20 12:14 PM.