Nope, not the goblin camp, though it'll catch up to you there too.
I've been doing quick, minimalist replays with just Gale and Astarion recruited, leaving Shadowheart to stand around outside the goblin shindig; if you do that, when you eventually transition either to the Mountain Pass or out of the Underdark, the artifact will catch up again (without SH this time).
