Fire Elementals, fire spells and weapons with bonus fire damage worked for me. The fight still takes really long, but it's not like he can damage you a lot. If you have the money, I'd suggest hiring a mage or two from your Homestead to speed up the fight.
Luckily the two-handed warrior companion (from town) had a trident with fire damage. Still took a LONG time though.
But now I know the trick with the wand. I don't remember the druid telling me about the wand, but that's probably because I lost a game of RPS.
