Did you double check the quest log to verify what you did in each of the situations the totem asks about?
Did you search around the edges of the village? There may be another way to leave, or at least get away from most of the goblins. I don't recall how it works before talking to the totem...
There should be a trap door to the far east of the totem (at x:213 Y:241) leading to a cave.
Do you have a recent save you can reload and just avoid the goblin village until you are a higher level?
Even if you try to leave without permission and trigger combat, you can have the other characters flee, as long as they are not too close to an opponent. If they are, and can not move clear, have the first character that teleports out take both teleporter pyramids, then drag and drop it on the first character in the combat turn order, for them to use to escape, and if required, repeat for the others.
