If I remember correctly you should be at a higher level when doing this quest. If you turn the quest in to Louis, the bandit camp is empty though you can still get to the temple. I think I was level 15 or so when I did the temple quest.
Reality is but a shadow of one's imagination, is it not?