I kill Gur in the village, when we refuse to help them, and they attack us, mind you, simply for refusing to help. After that, of course, no one comes to Cazador Castle.
They are still innocents. Of course they get mad, especially if Astarion is with you. He is the one, who kidnapped their children for Cazador.
"We are all stories in the end. Just make it a good one."