The bandits outside the camp will all attack you, regardless of whether you have the quest to kill Jagon, or even if you helped some of them at another location. As you found, you don't get a chance to use the password until you reach the actual entrance to the camp.
Welcome to the forum.
