I think you're probably thinking of 2 (or even 3) different chapels.
The one with the bandit camp bandit is in the wilderness. You can always re-visit using the trapdoor in the Goblin camp near Jackson farm.
There is one next to the city church you gain entrance to with a specific side-quest (not ratting out the guards in the tavern being drunk). It locks if you tell Louis about it though.
The main quest "chapel" is the third though, enter the church in the village and in the back there's a huge door, go through there, and find the ghost in question.
Good luck!
