Dont think the problem is so much that there is height itself, but the fact that the heights are so disproportional when engaging.
What i mean is that a goblin on top of mount everest shouldnt even be in the battle if your party is at sea level. If anything a good fix would probably be to make the range of bows and spells double their normal range at height or 1.5, and fix the cameras. As it stands its pretty much limitless in an up and down scenario. Its a really nice benefit when you push something off of a 100 foot cliff - which should really be an insta kill.
IRL a fall from anything 3X your height is considered life threatening. Apply that to a 4 foot tall goblin and throwing him off of the 20 foot high building in the village should be pretty much death or maimed.
Its also a pain in the ass at the goblin camp, but considering they are firing from a fortified position that was intended to be used by elven archers i guess it makes sense in that application.
Maybe the answer is not to apply that logic across the board in every encounter but just the ones that make sense. Id hate to see the majestic crumbling walls of the temple of selune reduced to 10 foot high speed bumps.