I kill Gur in the village, when we refuse to help them, and they attack us, mind you, simply for refusing to help.
After that, of course, no one comes to Cazador Castle.