Some ladders (such as the ones around the bridge to the githyanki patrol) won't let you climb them.
I've also noticed this with several ladders now, and it's always the really long ones like in the goblin camp and zhent hideout; in combat, out of combat, it doesn't matter. A character will climb to the top of the ladder and teleport down to the height of the bottom of the ladder. It makes some paths inacessable for characters that don't have a good enough jump.
