I have activated all 4 totems, but no fight happened? My quest log said that "Four elven challengers attackers rose up to attack us"

But nothing happened..?