This quest is heavily bugged, you need to speak to the totem when all 4 idols have a glowing aura around them.

The only idol that keeps it's aura after activation is the Spring idol, others you will have to activate fast and speak to the totem.