Reloading the game with invisible familiars (Imp) seems to cause them to still be on screen, but you are unable to command them in any way. Their icon no longer appears above you. I'm a Warlock if that makes a difference.