There's nothing wrong with putting points into the elemental damage slots but it really doesn't add all that much damage to be significant. I did that in my first game and it did help me out a little but I was beating myself up for not putting the points into something like deathblow which would help you with the holy smite thing that you were talking about. Personally...I would go with deathblow and flash attack rather than the elementals but that decisions is completely up to you but you should put points into the spiritual damage slot.
