Assume a 65% chance to hit and the same to knock them prone.
5 normal attacks with 65% chance to hit
5 [1d8+3=7.5] attacks at normal are expected to deal 5*0.65*7.5=24.4 damage, plus the additional 1d8 crit damage 5% of the time -> 24.4 + 5*0.05*[1d8] = 25.5 damage
One successful trip followed by 4 attacks at advantage
4 attacks at Advantage are expected to deal 4*(1-0.35^2)*7.5=26.325 damage, plus the additional 1d8 crit damage 9.75% of the time -> 26.325+ 4*0.0975*[1d8] = 28.08 damage
The (1-0.35^2) term represents the chance that you don't miss on both d20s.
Accounting for the 35% chance we fail to knock prone
You have a 35% chance to not knock the target prone, meaning you'll make 4 attacks at normal with an expected damage of 4*0.65*[1d8+3]+4*0.05*[1d8] = 20.4, where again the second term is accounting for the extra crit damage.
Thus, our total expected damage for an attempted trip and 4 attacks is 0.65*28.08 + 0.35*20.4 = 25.4 damage. This is barely LESS than making 5 attacks at normal.
Again, I'm not accounting for crit shenanigans or sneak attack being allowed on only the prone enemy. And of course I'm not taking into account the benefits of reducing an enemy's speed.
And the math slightly chances for different % to-hit, but 65% is roughly the expected for 5e. A higher chance to hit makes things even worse typically.
Edit: The math significantly changes if the following attacks would deal more damage. E.g., the fighter (dealing 1d8+3 damage) knocks an enemy prone, gets 3 follow-up attacks, and then the rogue (1d6+3+3d6) also gets to have a higher chance to hit. In this scenario, the prone+4 attacks deals 33.74 damage and the 5-attack-only deals 32.15 damage. It's better to knock prone.
Edit2: Further building on the above sneak-attack scenario. If there are a total of 2 normal (1d8+3) attacks + 1 (1d6+3+3d6) sneak attack we're back to it being better to simply attack than to shove prone + 1 attack + sneak attack.
Last edited by mrfuji3; 02/08/21 10:23 PM.