It sounds like a problem with the game itself, rather than you doing anything wrong.

Does hitting E again do anything? I'm pretty sure you should be able to get off a ladder by selecting it again, or hitting the 'cancel' button (don't recall exactly).

Have you installed the patch?

Maybe a clean un-install and re-install of the game an patch would help? (ie if the problem was a corrupt animation file, or something)

I don't recall anyone else reporting this problem...


First Aid Kit for Divinity II

First Aid : when Div2 does not start or crashes