Follow the 'Infiltrating the Immaculates' quest and you will get access to that area. Specifically, you need to get into
the cellar in Loic's chapel in Silverglen. Follow the quest or Pickpocket him.

Alternately,
getting a character arrested in Silverglen will get them sent to a prison in that area. If they are seen escaping, everyone there will turn hostile.