Yah, the normal animal familiars should be treated as neutral by enemy mobs. I had the bright idea to send my crow out and scout the goblin ambush to see if I could find a back way in, but they aggroed the raven and I was stuck in combat even though none of the goblins could see my party.

Very annoying. Scouting should be part of the familiar utility.