Technically, the imp isnt necessarily a "real" imp. It isn't the final form of some evil baker or low level mook. Find familiar puts a spirit of your choice (fey, celestial or fiend) into a form. Normally you only pick beasts, but the Pact of the Chain lets you pick other forms. So you could have a fey imp, a fiendish pseudodragon, or a celestial quasit (just kidding, no one ever picks quasit).
Sounds all like wishiwashi something to me.
Or rather that there are just many different types of Familiars. But it is not because of some "superior Familiar magic" that they take a form.
More like many beings
which are all real and complete in what they are can 'become' a Familiar.
There is no Magic that makes a Spirit look like a Fiend. ( Appart from Polymorph. )

If you pacted with a TRUE Fiend, probably not the little sh*t we can summon but a greater Devil even, they will simply allow you to call forth one of
THEIR servants as aid for yourself.
This is how I have seen this little Fiends all this time. Of course this must not necessarily be what has occured.
Maybe we asked such a little, unimpressive Fiend himself, with a deal that let us even keep our Soul just because a little sh*t Devil like this is not impressive enough to sell our souls too.
Cause what can he give, huh?
I imagine this totally lowclass tiny Devils to be glad if you just offer them to be able to kill someone, aka doing Evil stuff for an Evil alignment creature, just to be willing to answer your call.
But Warlocks seem more powerful. They get access to abilitys far beyond just summoning a little Devil.
I imagine that the Warlocks we can play, are all at least in carhoots with a Cambion at least.

Juicy details asside - I doubt that Paladins, Clerics or Druids know about any of this and why would they? As they are opposing Deals with the "evil forces" by all means, right?
They wouldn't know why a creature looking 10/10 like a Devil, would follow you around without trying to slice your throat open immediately.
So everybody might assume you are one of this infamous fools making pacts with Devils and allowing them more influence and power of the people o Fae'run in one way or another.
Someone walking around with a Devil... HAH. As if they would not ask questions.
But hey the whole being a Drow part is also mostly unanswered and leaves more to desire.
Drow are infamous themself for being a type of super sketchy, evil Elf.
The few little mentions of it here and there are truly... cute.

Gives me hope for the final version.
Being of a highly infamous Race or walking around with a very infamous creature is not truly mirrored in the current version of the game.
I dunno about Tieflings but the Druids, who host many Elves as their members, they should be way more wary of Drows.
Ask the dwarfen Poison Ivy. ^^