They were spread out a bit when I did that part of the game, though still all in the tavern. If you go stand beside the guy outside, can you block his path away from the tavern, and try to herd him back in? That would be a PITA, but worth a shot if you don't have a recent save you can reload.
