Every time you specialize your character, any advantage you acquire is at the cost of a disadvantage elsewhere; the bigger the advantage, the bigger the disadvantage should be.
Actually no. Any spec point you have are 100% advantage.
Find a single specialization that gives you a disadvantage right now in DOS2. There are none. You may choose not to have this or that but that is in no way I pick Warfare so I have -5 in intelligence. ALL specs are PURE BONUSES as it stands (skills and traits are different.)
They don't always work as well every time (Warfare is useless when fighting high HP low armor enemies), in some situation they might not work at all, but no other spec is forcing you to use it to your detriment.
But when you have no control over a specialization that can screw and you have nothing you can do, sorry, but that doesn't work.
Now as it was pointed out you can prevent it with decaying touch. You can stop it, therefore I am good with how it is.