Unless you hit Ifan with area effect damage, etc, he shouldn't turn hostile to you, regardless of guards starting combat nearby (he shouldn't react to combat, either joining or or running away like a civilian NPC). I just tested it to double check (in case the collar being removed was the crime the guard reacted to, having Ifan join by systemics), accidentally killed him, resurrected him (developer build), escaped combat and returned, and he was neutral again. An NPC challenged me about a recent murder, called for guards and started combat, which Ifan ignored, and after it was over he suggested we team up when speaking to him.
Did you get any earlier warning for trespassing or vandalism, etc? It is possible earlier crime warnings could have caused Ifan to join combat when the guard spotted you. I tried moving / opening the offerings around the waypoint to trigger combat, and while Lohse did join, Ifan did not (likely due to being farther away).