When you first arrive at the gates of Fort Joy you can actually skip the event by taking the ladder to the right of the stairs. You can come back later and fight Dallis, Alex, and everyone else present there -- so long as you don't escape Fort Joy. In my experience the hammer didn't always drop -- I'm not sure if it was because specific variables of the fight had to be met or if it was pure chance. (other times it just looks like the hammer drops through the ground)