If the game were to go hyper-realistic with tactics you'd need a revolving sun. Having the sun at your back is a huge tactical advantage as your foes have to deal with glare in their eyes.
It's exactly as you say. High ground is advantageous mainly because it allows you to ignore cover and more easily shoot in to masses of the enemy instead of enemy+ally groups. Also, it takes longer for enemies to reach you if they have to climb a hill vs run down a hill, so you have more time to shoot them.
And 5e already has a system for cover. Shooting past a character grants the target half cover (+2 AC, +2 Dex STs). Hiding behind a big obstacle or merlons (those protrusion on top of castle walls) grants three-quarters cover (+5 AC, +5 Dex STs). Notably, neither of these scenarios grant Advantage, which means they stack with other sources of Advantage. On a un-crenelated hilltop where there are no merlons to hide behind, the bonus for height advantage should then be +2.
Adding to this, getting the high ground also meant your units were at the battlefield first. So you would have more time to set up traps, learn the lay of the land, etc.
We can't logically give all the benefits of being the first-mover to "high ground". We have to separate the two.
Which goes to Mr. Fuji's post. If you put two equivalent armies in a vacuum and one starts on higher ground...
The benefits would be:
- The moments the enemy had to put down weapons to climb (climb speed)
- The reserved energy from not having to run up a hill (rough terrain)
We also can't give the benefits of having a castle to high ground:
If both sides just stood their ground and fired arrows at each other, their accuracy would be equivalent. It's why prior civilizations built forts and castles. You could design vantage points that could provide protection for your archers (Literally some castles had trapezoidal windows). It's also why Fortnite became a popular game, being able to create your own cover on-the-fly is more strategically interesting/advantageous than shooting at each other in a vacuum.
Go back and play old school Quake or Halo and high ground was mostly exposure. It was really easy to see a player standing on a structure. What mattered more was who saw the other first.
Play nerf swords and such with some kids. See what I mean. It is SO much easier to hit one of the 5 kids running around from higher up than on the ground. Likewise, when even 2 screaming kids attack you with melee weapons, it is really hard to block all of them. Thus, advantage makes sense to me.
You're an adult versus children, you have a longer reach. Which is why taking polearm master and a glaive is a popular strategy in 5e.