That being said, the high ground bonus is pretty powerful. The fight against the hob goblin guy in the temple of seluné is a good example. On the ground I barely came out of the fight alive. But When I climbed up to the ramparts above it became almost trivial, especially when I had both my ranger MC and Gale in the party. The worst that happened was one of my guys getting knocked off, but even then they survived well enough.
Now flip that with the Harpy encounter near the Grove. The harpies start out with the high ground bonus and all of your guys start on the low ground. The fight is by no means horribly difficult, but the combination of their high ground bonus, and my melee characters having to trek across the battlefield to reach the harpies, made it a challenging fight. The high ground can definitely be a deciding factor in a fight. I would not oppose Larian trying to tweak it a bit if they desired.