If you decide to keep adventuing, it can be seen as a return to visit the city and see how it has been fairing in your abscence. Same with those who decided to travel to Avernus with either Wyll or Karlach. Simple fix, really. You can have a rented home in Baldurs Gate while you are visiting to see some of the new improvements you can decorate.
