It bugged for me, too--no matter how close to the ladder I went, they never attacked. Worked as intended on a previous (since discarded) playthrough, so not sure why it's not working now.