I think the way the mechanics work is not as straightforward as one might think. I can't really prove it without further study, but I think the amount of interactions you have with them, as well as the order in which you talk to them if more than one of them has an exclamation mark, affect how they treat you