Did it. Finally!! Took two hours because I probably didn't have my party built or buffed as properly as I could have. But did learn some things and some things here were useful.

1. Stay well away from him with your party till the very end when he is nearly dead. He is deadly until then.
2. Save often. Save after every moment when you either hurt him or stun or freeze him.
3. Drain Willpower and Soulsap are your friends just keep pounding him with them.
4. I came into the fight with 20 stun arrows for my ranger. After he was soulsapped and drained I just kept hitting him with those and ice shards. Took a long time but did the trick (knockdown less effective and seems to work only once out of every three tries).
5. Keep a bloodswarm on him. He seems to be able to miss their attacks less frequently than other summons. I don't know why.
6. Do charm that spider.