Being tossed of from high altitude is and should be a threat, which is why you want high athletics score to resist.
The problem is that it's not resistable at all, at least according to the lack of rolls in the combat log. You seem to automatically succeed.
Shove MUST be a CONSTESTED Athletics checkUnless my playthrough was bugged, I also believe Shove is currently a Bonus Action or Free Action, whereas it should be an Action. I was able to shove after casting an Action spell or using a scroll.
It's worth noting that you can trivialize fights with this ability and it seems like without a significant revision, the ability will greatly limit the environmental design space for high CR monster encounters.
For example,
I one-shot Auntie Ethel by shoving her into the pit.