I teleported him out without anyone noticing, but after I spoke to him in the cave whenever I would get close Griff's gang would attack. I could walk around the camp before I spoke to him fine, but not after. It was frustrating.