This is mainly to demonstrate that the following statement is untrue: 'Well in fairness Armor and Vitality are total garbage and spending 0 points in them probably makes you stronger overall.'
No I stand by the statement 100% because math.
Here is my Lhose at level 6, wearing the best equipment I had on me which wasn't in use by someone else:
INT = 19
CON = 12
MEM = 12 + 2 = 14
WIT = 12
HP = 209
P. Arm. = 57
M. Arm. = 53 + (53 * 0.07) = 65
I took the Talent "All Skilled Up" to give me an extra Ability point, bringing my total free points up to 4. I had 1 point in Magic Armor already for a 7 percent boost. Putting those 4 ability points into...
...Vitality boosted my HP by 14 points from 209 to 223 HP.
...Physical Armor boosted my Physical armor by 8 points, from 57 to 65.
...Magical Armor boosted my Magic armor by 4 points, from 65 to 69.
(I'll note that 1 point in CON boosts HP by 17 points. In other words, 4 levels of ability points into Vitality is worth less than 0.5 levels worth of attribute points invested into Constitution.)
I then found an encounter with Level 6 enemies and I sent Lhose in alone. That was not to see if she was to survive (that would be impossible), it was just the fastest and simplest way to get some numbers on the enemy damage. [Square Brackets] denote DoT/Surface damage, which I'm not counting for the average.
Test one (+14 Vitality from base)
The enemies did...
...physical damage of 47, 79, 19, 22. Average = 42.
...magic damage of 26, [31], [40], [33], 62, 16, 54, [20], 46, 62. Average = 44.
Lhose survived until her turn without being CC'ed, which allowed me to boost her HP with Encourage and heal herself with Restoration so she could last a bit longer.
Test 2 (+4 Magic Armor from base)
The enemies did...
...physical damage of 54, 74, 20, 78, 97. Average = 65
...magic damage of 26, [33].
This time they used different abilities and Lhose was knocked down and stunned, getting no turns.
Test 3 (+8 Physical Armor from base)
The enemies did...
...physical damage of 45, 79, 21, 75, 130. Average = 70
...magic damage of 26, [33]
This is with 4 ability points invested by level 6 (5 total in the case of Magic armor). Averages aren't going to be that useful since there's usually 4 people in a fight, but just look at the raw damage numbers. 14 HP/8 Physical armor/4 Magical armor.
In conclusion, points into Vitality and Armor only make a difference if you have so much already that you don't really need it, and therefore, those skills are not useful to invest in.