That was the math. A roughly 50% swing going from Advantage to Disadvantage. You think that is too drastic, so what are the solutions?
Well, we can change Advantage/Disadvantage to work differently. Please trust me on this one, that is NOT the right answer. The rules of 5e rely heavily on the Adv/Dis system. Changing how it works is the equivalent of removing 2 tires from a car. It might still 'work' but you've thrown away everything else it can do. It is a beautifully elegant system that removes so many small factors from having to be considered.
Alternatively, what if we just removed some of the factors *granting* a character Advantage or Disadvantage? What if having even just slightly higher elevation didn't grant Advantage? What if the range for an enemy to "Threaten" your character was much closer, meaning your Ranged character wouldn't have Disadvantage to attack with their bow unless the enemy was right up in there face?
Suddenly the swing isn't so drastic. You go from Disadvantage if an enemy is right up in your face (1d20+0) to Normal if you're attacking from range (1d20+5).
There are a ton of other factors that having a height advantage can be used to help players/enemies. But, by just removing Advantage from having a slight height advantage (something that is *not* in the rules of 5e by default), and making it so enemies only Threaten you when they are able to make a melee attack against you (which is how the rules of 5e work for imposing Disadvantage on a Ranged attack) we've solved this huge problem you're experiencing. Basically, by following the rules of 5e, you've solved the problem.