If we take a non-Variant-Human situation, you'll either take the feat or get an ASI at level 4. With a +4 in STR:
- A normal hit does 1d12+4 (10.5) damage and has a 75% chance of hitting, for expected damage of 7.88
- A normal hit with Advantage has a 94% chance of hitting, for expected damage of 9.87. 25% increase over normal
We see that GWM (with a +3 in STR) is still superior: +11% damage at Normal, and +38% damage at Advantage.
Also (but no less important), you have to consider damage break points. Dealing 1d12+3+10 damage is *vastly* more likely to kill an enemy than 1d12+3, which
a.) prevents that enemy from taking additional turns, significantly affecting the outcome of the fight
b.) procs GWM's BA attack
If 5e was balanced so that you're typically expected to have a 50% chance to hit (e.g., AC 16 enemies at level 1) then the math is much less favorable to GWM. Alas...
Your math is correct in your particular case (for greataxe though, greatsword would average 7 points of damage, not 6.5), however there are couple more things to consider.
1)
In BG3 your base damage will be higher than in PnP, mainly due to magic weapons that add damage on hit and/or dipping weapons in fire, coating them with poison. The higher your base weapon damage is, the greater the effect of lower attack hit-rate is, and the smaller gain +10 to damage is.
In case of level 4 Lae'zel that uses Everburning Blade with 75% hit chance the difference between ASI and GWM would be:
-
-5% damage per round with GWM on normal attack,
-
+18% damage per round with GWM on attack with advantage.
2)
While calculating the effect of getting bonus attack on reducing enemy to zero hitpoints is hard, GWM feat also gives you a bonus action attack on critical hit. So we just need to modify expected damage by +5% in case of a normal hit, and +9,75% in case of advantage.
So in your given example (Great Axe base damage, 75% hit rate) the differences would be:
-
+12% damage per round with GWM on normal attack,
-
+49% damage per round with with GWM on attack with advantage.
While the difference is minuscule for a normal attack, with advantage it's definitely notable.
3)
Combining considerations in points 1 and 2 - Lae'zel, lvl 4, GWF, Everburning Blade (or any dipped/coated 2d6 weapon):
-
+0.1% damage per round with GWM on normal attack,
-
+28% damage per round with with GWM on attack with advantage.
So even in 75% hit chance case, which should be highly favorable to GWM the difference is not big enough to consider the feat OP. At 60% the gap would be -9% and 6% respectively and at 45% the damage would be better without power attack on and difference due to -1 Strength would be -12% and -5%.
In conclusion in BG3 specifically GWM is not overpowered choice for two-handed fighting style, but it's a good feat that competes with ASI. Would I take it for Battlemaster Lae'zel? Probably not on level 4 or 6, but on level 8 most likely. For a Wildheart Barbarian? As soon as possible.
If Larian adds PAM, that works the same way it is used in PnP and/or Variant Human (no sacrifice on Strength score for getting a feat) then my opinion would change.
That all said, if OP asked are two-handed weapons too good compared to two weapon fighting or swoard and board, my answer would be yes, especially after getting Extra Attack. In addition the feats the other styles get are a lot worse than GWM.
