I ran into this problem as well. I just fast-travelled out of the fort. I doubt they'll let you do that once Early Access is done though, I'm guessing they'll limit fast travel at some point.