Agree with others that the minimum height for advantage should be increased. You should need a huge height difference to gain advantage, such as:
>the top of the gate in the Githyanki fight
>the rafters in the Shattered Sanctum
>the cliffs when fighting the Duergar

But I'm pretty sure in the Githyanki fight, I could get height advantage just by standing on the little rock to the east. That just doesn't feel right in my opinion.