In previous editions of DnD, Magic users would often have fiends as familiars, and not necessarily evil magic users either. The assumption often is that you are in control and that creature is obeying you. In 4e, for example, I had a bound demon as a familiar and at a low level would summon a dretch to take care of my enemies. But at the time I was not evil (yet), and I had those things under control. Similarly, in Faerun, magic users employ lower level fiends often. They are easier to bound and/or trick into servitude, and there is the assumption that they can not do as much harm as a higher level fiend. Many of these magic users may be evil, but some are also good and use their conjuration/enchantment skills for heroics. So while a Druid or Cleric may not like it, it may be uncouth of them to jump to the immediate conclusion and have worse repercussions attacking the mage that is helping save their home or is keeping the devil weak and bound. That being said, more dialogue of discomfort or concern would be good, or even options (that might not necessarily be good aligned) that open up purely because the familiar is a fiend instead of other options.
And in current 5e, surface drow have become more and more of a thing so they are not considered as sketchy as they used to be as there are more interacting with the people.
