This is so annoying. I wasn't sure if it was only related to ladders or could also be caused by more general blocking of paths. I had it happen to me on the cliff-face to the right of the main gate to the goblin camp - and I don't recall anyone using ladders before. If anyone is impatient, f5 f8 also reverts things back to normal again.
