As others stated, I just sneaked past him into his private room, and engaged from there. You basically can attack the ranged ones right away, and the melee will be further away from your group, giving you a couple of turns before they reach your position.
If you engage from the door, you'll be in a bad spot. Not to mention, you probably won't be able to trigger his conversation, instead a guard will just alert your escape and the fight starts.
I killed him at lvl4 if memory servers, and was quite easy at that point. Didn't even escaped the Fort yet.
