I was able to get it peacefully by dropping a midnite oil directly on it and then two more leading to the fire to the south of it. This started them all on fire & the totem was destroyed. I made sure not to do it while the one goblin was right there to avoid hitting him directly. He did walk into the fire and take some damage but it didnt agro him.
