Ive killed all the guards at the quarry including damon(if that was his name) and I get killed when I enter the front door by the 2 guy, all I can do is leave as soon as they head for me, am I missing something
No, these guys are just very high level and it is best to come back for them a lot later in the game.
