Who cares if a high level wizard sometimes misses? Is that an issue? Are high level wizards always supposed to be able to hit?

Maybe the wizard has something to get advantage, to thus cancel out the disadvantage.

This really doesn't sound like much of a problem.