For me, the most difficult fight was indeed in Shar's temple, more difficult than Raphael or even the final fight.
The thing is, it's not the one OP is talking about, it the one in Act 3 which resolves Shadowheart's story.
Nevertheless, on my second playthrough even this fight was easy enough.
