No, you don't need to kill the goblins first, but they will go after you when you destroy the totem.
I snuck up to it too, hit it with my sword and it was instantly destroyed.
Try to see if the totem has particular resistances. I can't remember what sword I used. Maybe the totem had a weakness to whatever I was hitting it with. I can honestly not remember.