did you damage the war drums? that upsets them as of this patch
there are also restricted areas that can cause a fight i.e, the halfling merchent and the priestest home both get you in trouble unless they invite you there but only if someone sees you are in there... its like the army, what isn't reported didn't happen so its fine
