I used the set items, and the bonuses they offered were HUGE and worthwhile - so worthwhile that I'm planing to make a character that never uses set items.
A very good plan I think to make the game for yourself a(real) challeging in the meanwhile (I hope) Larian make a
crazy Nightmare mode
(but do no change anything on the loot!)