When you are close to the ladder, on the correct side and with the crosshairs pointing at it, a message should show up on the screen (similar to doors, chests, etc) indicating it can be selected. Doing so should have your character walk up to the ladder and grab onto it (sheathing any weapons, if required), at which point moving forward/back should have your character climb up/down the ladder.
