As far as I can tell... No.
One of the Druids has Truesight so invis/mist/darkness doesn't work.
Moving it at all, makes them run towards it.
There doesn't appear to be a way to get it without abusing teleporting to camp.
That said, there is a less timing dependent way of doing it:
The way I usually do it, is by having my party stand on the slope below it, then having Lae'zel cast Mage Hand and then have the Mage Hand throw the idol down to my party where I then pick it up and go to camp. Touching the idol with Mage Hand makes the Druids get miffed at the hand itself rather than you and on the slope you're out of their line of sight so long as you wait for that one Druid to not be standing at the top of the slope.
