They don't attack right away when you pick up the thing, you have to get close to the ladder. If this doesn't work, then it's a bug I guess.

"We make our choices and take what comes and the rest is void."