You need the quest first, then you can open the secret passage in the ruined chapel (west of the village, north of the river, in the initial area you have access to in Broken Valley). It will be awhile after that before you will be able to reach the bandit camp (groups of bandits on the way are level 9; for one area in particular in the camp you would probably want to be at least level 11 or 12 before attempting, or possibly at level 10 with the help of the creature or a summon).
