Fire Elementals, fire spells and weapons with bonus fire damage worked for me. The fight still takes really long, but it's not like he can damage you a lot. If you have the money, I'd suggest hiring a mage or two from your Homestead to speed up the fight.