This is true! But if I open the chest, in private, how do these people know? I understand that they would be angry if they DID, but for them to react with aggression implies they are mind readers. Other similar situations (eg, stealing the statue from the druids) does not cause such an immediate response, and you are able to talk your way out of it. I'd expect the same here.
