All of the bandits on the way to the camp will be hostile. The sentry at the entrance itself (in a small shack) will demand the password and let you in if you provide it; everyone inside will assume you are a new recruit. Make sure you complete everything there before you report their location (though I think you can leave the Temple of Doom until later).
