Yes I have been neglecting my defense, because I can get away with that this early in the game. At that point my Inquisitor had 0 Bodybuilding and 1 Willpower (plus 1 from a Trait), but after reaching Level 13, I will have enough ability points to have gotten all my skills as high as I need them, and will be able to dedicate more points towards those abilities. (My Inquisitor was sitting on 5 unspent Ability points at the time of the video)
At the moment, my primary defense is shutting the enemy down first.
Another interesting fact is that I COULD have spent a point onto Pet Pal for my Inquisitor at level 7, since no one else has it, but I haven't bothered - and yet my level seems to be higher than expected - I haven't finished Hiberheim yet.