If you bear the absolute brand they should at least be neutral with you.
Hmm, have you ever branded the whole party?
I do seem to remember killing Ragzlin without aggroing the whole camp. I only managed it once. And I only branded the whole party once.
