It seems that every time I level up the negative damage modifier on skills increases so that my skills, in this case marksmen/finesse, do less damage unless I use my points to increase my finesse to which they go back normal damage. I guess I didn't notice cause on the rest of my party because I kept getting better weapons, but I've been using the same crossbow for a while.