Most of the castle guards turn hostile if the duke attacks you, and you can't kill the duke, he's one of those invincible characters, you can get them to the lowest hp then they stop losing them.