It seems that every level, the 'baseline' amount of primary attributes needed to have a 0% damage bonus (eg neutral) go up by 2. Having points above or below this baseline gives a bonus/malus, and this amount decreases by level, possibly scaling with your % difference from the baseline.
The net effect is that you need to spend both your skill points per level just to stay afloat of the baseline. If you created your character with a initial bonus (eg 13 strength vs 12 baseline gives 10% bonus damage or whatever), then it will decline over time even if all points are put into it (at later levels, 23 str vs 22 basline may only give 3% bonus).
This makes hybrid classing impossible, and investing in wits/memory/vitality a permanent damage debuff.
