I'll give it a go on a restart on Hard. Pretty stupid that I restarted rather than keep my old game and just switch difficulty, huh? DOH!!
Not stupid . I would do the same as you.
Maybe it's strange, but in some way it feels to me my current hero playing on normal or hard CAN'T change his/her "standard/level of challenge" like it is a peace of cake to do that out of nowhere ...
It's, for me, a part of a roleplay (role) that I prefer to start again a new hero rather than simply change the challenge mode of my current hero, especially from the hardest level to a lower level . Vice versa, my hero can live with that.
But of course it's no shame to change the difficulty in one playthrough .
And :
with specific enemies I always must be very carefully planning ... or I die ... like with Zagan !! Even on "casual level" I've problems with that son of a bitch

... so change the difficulty (lowering) for Zagan is out of the question for me . If I'm clever enough I beat Zagan, even on nightmare mode !