This issue was due to being able to spawn totems outside the sight range. That was fixed, but not in time to make it into the last update.