I never even knew, that was possible. In case of the goblins in the Blighted Village, you can just use the secret passage near Scratches location to surprise them and have an easy fight.
I agree, that this shouldn't be possible. I think, it's OK, to distract a merchant (but even then, they should be allowed to role perception with disadvantage).
But to cross nearly a whole village and attack without provoking an attack, is just stupid. As a DM, I wouldn't allow that in game (maybe one sneaky character, but not a whole group).
