Try returning to Broken Valley, and then go back to the fjords. A couple people have run into single opponents that wouldn't take damage on this path; one jumped down the mountain, and was able to kill the problem opponent when they eventually worked their way back up, another left and did stuff on sentinel island, and the creature was vulnerable when they returned.
If an opponent is at the limit of their range (they will not follow you more than a set distance from their starting point), they will not take damage. At that spot on the trail I couldn't do any damage with a bow at first, trying to attack through the bushes there, at the edge of my range. Once I moved a little closer and got the attention of a few of them, I didn't have any problems doing damage (though I had to withdraw a couple times to heal).
