I had the same problem, but after returning the chest: for no reason I was attacked when I moved away from their leader (the bridge closest to her)