Originally Posted by Elessaria666
Best way to avoid it for me has been to click on ground at the top/bottom of the ladder/cliff rather than the ladder/cliff itself.

Same.