Thanks for re-stating your argument, I think that helps me figure out the core of what you're saying (though I agree this is more a thread for the general section). I think you have a good point about saying "it's fantasy so anything goes" isn't a great argument, but I also think that your base claim about what stretches the bounds of believability in the first place. I also think that a lot of those genre conventions are just conventions and not essential characteristics. I think your argument is better aimed at specific settings as opposed to fantasy as a whole. I think that on a broad level, because it's fantasy anything can be permissible, but to use your Duergar example, I'd also say that sure, anything CAN be possible but based on the established lore for this setting, that's not how it works. I think that's fair especially when physically duergar have one real visual defining trait.
I also think that the "It's fantasy" argument also gets used as much as it does out of frustration that somehow acknowledging identities and diversity that did not originally exist in the genre because it was established at a time when such things were not acknowledged in the society at large is somehow harder to accept than fantastical magic and inhuman entities. I'm sure plenty of people make the argument sincerely, but in my experience it's not the norm.