Shoving people off sizable cliffs, let's say ladder height, only tends to deal like 5 damage at most, and they just get up, hardly having lost movement. That's crazy good for a bonus action, but for an action not amazing. It needs to feel rewarding to sacrifice attack and take these opportunities, but not be able to shove everyone with anyone and have them fly like a rocket
Sure, it might only do 5 damage. But you're also getting the enemy away from you and your squishies. Realistically, they should also land prone. This would remove half of their movement, preventing them from simply climbing back up the ladder and attacking you on their turn. Spending an action to (deal a small amount of damage and) prevent an enemy from attacking me next turn seems like a fair trade to me.