It appears he's only there the very first time you make it to the grove. I think if you rest or leave before meeting him he leaves for the goblin camp.