That was the only passive damage skill (besides Augment Damage) that I did put points into. Eventually monster resistances will start getting higher (I'm pretty sure it can be resisted), but by that point you will not need the extra damage as much. I bought a sword from the traveling merchant on the road to the cursed Abbey with Spiritual Damage(3), then went back to clear the orcs around Aleroth... the spiritual damage was just about equal to the physical damage. <img src="/ubbthreads/images/graemlins/biggrin.gif" alt="" />
As a warrior, you should have plenty of spare skill points (I had 10 unused at the end of the game, put 5 into a skill I didn't use and maxed a few skills I didn't really need after getting to the wastelands).