Most of them are called Illuskans, or Northlanders, and are fair skinned. Calishites, Chultans, etc. are not. From what i've seen of the game easily more than half of the humans are from somewhere other than the Sword Coast. Just like there being hordes of tieflings, it makes no sense, only it's worse because there's no good explanation.
Not that I expect them to care or change it. "Representation" is mandatory in western culture nowadays.
Straight from the PHB:
Tethyrian
Widespread along the entire Sword Coast at the western edge of Faerûn, Tethyrians are of medium build and height, with dusky skin that tends to grow fairer the farther north they dwell. Their hair and eye color varies widely, but brown hair and blue eyes are the most common. Tethyrians primarily use Chondathan names.
So the inhabitants of the Sword Coast are actually too white. Need more brown people in the game.