I found a while back, there are definitely times you don't want your or Gale's familiar out. Found that out on the ship. Haven't had an issue with Wyll's familiars, but I usually don't have them out or I use the Imp and make it go invisible.