There are many ways to make this easier, the biggest being having your people out of range of him and the cambions by being along the higher left and right that is raised up and pelting everything with ranged attacks. Then when one runs up just push them off and rinse and repeat, or thats how I take care of them. This patch they seem to just stand around just under me after a push or messing their aggro with an Mage hand or Illusion, letting me pelt them with hex enhanced Eldritch Blasts.