I have experienced this problem as well.
In fact, both the the problem of characters breaking ladders while on it, and characters choosing to jump down when a ladder exists (such pathfinding, much sense of self-preservation).
The rooftops of the Blighted Village is an amazing spot to experiment with this, if anybody wants to try.
