I've had this issue too, and I had no familiars at either of those times.

-Better a witty fool than a foolish wit-