Hey OP you are absolutely correct. And kudos to all the others who've corroborated this point. Yes in RAS's books there are a number of drow hanging out on the Sword Coast, but they all took a very long time of traveling to get there, in many cases years of travel. And yes, portals are a possibility, but portals are not that common and tend to suffer a lot of oversight by the powerful NPCs of the Realms.
Bregan D'aerthe teleports all over the place.
I'm pretty sure they're still in control of Luskan to the north, which does steady trade with Baldur's Gate.