Best way to avoid it for me has been to click on ground at the top/bottom of the ladder/cliff rather than the ladder/cliff itself.