The idea that demons - demons - aren't necessarily evil has been done as recently as Buffy The Vampire Slayer and Angel (if there are media examples that date back even further, I missed them). Again, an interesting one-off subversion, yet in no way the end result I want to see in Dungeons & Dragons. Labels should mean something.
Firstly, I think that Buffy can't really be used as a recent example anymore, being 30 years old now at this point. Also I would argue that such ideas are older even than that anyway. For a very recent example that's closer to the subject at hand though, I'll point you to Wrath of the Righteous. There we have a demon that required divine intervention to even realize that good felt good, and then further divine intervention to fully purge herself of that evil instinct.
With regard to the trend in D&D, I think that in large part it's a consequence of just how long it's lasted. Very few products have anything close to a continuing narrative the way D&D has. A product can't exist and update for decades upon decades without things changing and growing. Orcs and goblins and drow and etc have been around for decades, and the longer they're around, the more they have to grow and develop to continue to be interesting. Because inherently evil or not, they're still thinking, sapient societies, and people want to know more about them, and writers want to write more about them. And you can't develop a villainous society for decades upon decades without that society growing more nuanced and multi-faceted. And when you're not just talking about one society but an entire species, things get even more complicated. Each edition is expected to have more and to be new and different in terms of lore. And when that applies to the villains, then you can only develop them for so long before they start to look less villanous. Pure evil mooks that you can kill without a thought only really work if they're one-dimensional to some degree, and you can't have an entirely one-dimensional race and keep them one-dimensional as a baseline in a game/setting that's gone through decades of change and growth.
Rhen there's also an assumption built into these fantasy worlds that goes unspoken most of the time; that good is superior to evil. That good can overcome anything and the axis of the universe on a base, narrative level will always lean towards good. I believe that that assumption, combined with decades of time to work, will very easily start to lead to people wanting to see members of an evil race being redeemed. Because if evil can't be redeemed, then it implies evil being greater than good. This kind of shift doesn't happen as much with something like dragons because with dragons people still view them as a whole species as opposed to seeing them specifically in groups based on color, so it's like "of course dragons aren't all evil. We have metallic dragons just over there." On top of that, dragons don't have a society the way orcs, drow, etc do, so they tend to get treated more like individual characters.