I made a post on this as well. I've discovered two things:
1) The imp familiar, if currently summoned, seems to consistently make allies a the camp become hostile
2) If you teleport to camp using a waypoint during the day (as opposed to using the campfire button), allies can sometimes be hostile
If it was the quasit familiar, then that would make sense. My main character had one summoned
