As a tabletop DnD 5e player, I couldnt agree more. Giving advantage/disadvantage because of elevation is the worst offender, I feel like I am playing "who climbs the highest simulator" and not a real tatical game.
Height is a tactic though, I like to see it stay perhaps tame the benefits/penalties.
cover rules from PHB,
Walls, trees, creatures, and other obstacles can provide cover during combat, making a target more difficult to harm. A target can benefit from cover only when an attack or other effect originates on the opposite side of the cover.
There are three degrees of cover. If a target is behind multiple sources of cover, only the most protective degree of cover applies; the degrees aren't added together. For example, if a target is behind a creature that gives half cover and a tree trunk that gives three-quarters cover, the target has three-quarters cover.
A target with half cover has a +2 bonus to AC and Dexterity saving throws. A target has half cover if an obstacle blocks at least half of its body. The obstacle might be a low wall, a large piece of furniture, a narrow tree trunk, or a creature, whether that creature is an enemy or a friend.
A target with three-quarters cover has a +5 bonus to AC and Dexterity saving throws. A target has three-quarters cover if about three-quarters of it is covered by an obstacle. The obstacle might be a portcullis, an arrow slit, or a thick tree trunk.
A target with total cover can't be targeted directly by an attack or a spell, although some spells can reach such a target by including it in an area of effect. A target has total cover if it is completely concealed by an obstacle.
And yet enemies can still have advantage shooting through a crack in the wall from on top of a building personally experienced this during the fight in the goblin village.
Personally in my games, my DM would rule something like that with disadvantage.
