I've found it challenging enough to play on Normal difficulty with my recent builds (Magic Missile Mage, Two-Handed Sword Warrior, Ranger), but I've occasionally dabbled with Nightmare mode (I've sometimes played it for some time before realizing why things were so hard).
The other day I did a run through the final escort mission on Nightmare with one of my characters - to prove my point that if you're complaining about the escort mission, you're playing it wrong. It wasn't terribly harder than my Normal run through the game. The final boss however, did wipe the floor with me.
I might change it to Nightmare mode for a dual-wielding character, to counteract how powerful Dual Wielding is.