Same sort of issue in the goblin camp where you can break the wall to get into the temple. Even if you're inside a silence bubble, it wakes up the drunken goblins.