I'm barely one meter below the enemy, which means my attack is at disadvantage. This isn't even enough of a difference for a -1 on my attack roll in a "flat bonus" system. I do think a flat bonus system is the better solution, with +1 for height difference of 5-9.9 meters, and +2 for a height difference of 10.0 meters or more. No flat bonus higher than +2. That would still provide some benefit for height, but not one which is super strong.