That sounds like you've come a cropper of the bug introduced in a recent patch that causes offence flags to get set on your characters and not cleared - the game reacted as though you'd committed crimes or antagonised npcs each, and never clears those flags, to simply walking by some NPCs will cause affinity losses, and eventually they'll attack you, as though you'd hit them a few times.
