A part of the problem is they setup shove so that it always succeeds while you are hidden, and when you are fighting only one creature it is fairly easy to move out of it's sight and bonus action hide...