I have basically been putting all my Combat ability points into the Skill category I've been using because other than that, the only thing which seems worth investing in is your weapon specialization.
Of course, that makes sense since the Combat ability points go to [Weapon Type] / [Skill Set] / [Defense]. It's really just that the Defense category is too underwhelming compared to points in the other types.