Looking around, this seems to happen (to me at least) near Baldur's Press. Having someone take me to camp and THEN loading in goes perfectly fine, and I can load into Baldur's Gate from camp instantly without any issues. It's just trying to load for the first time directly into the city that causes an issue.
