1: Get a duergar in your party, you can hire one from Withers if you're not playing one.
2: Level them up to level 5, class and equipment are irrelevant hete.
3: Activate the Invisibility cantrip and separate from the party.
4: Walk unimpeded until you get to the fast travel marker.
Invisibility lets you bypass that encounter conpletely, and the duergar version is both and unlimited outsidw of combat, so that's a win button solution for you.
