Our game would sometimes work if we continuously restarted, but the loading screens were so long and it would only "fix" the issue sometimes that it didn't feel worth it.
We did try different variations of host loading in first with the whole party at camp and then the guest joining. I usually didn't have issues (as the guest) loading into camp (either the Lower City outdoor camp or the Elfsong camp) but leaving camp was a mixed bag. Hitting "Leave Camp" would usually make the issue happen and the game lock up, fast traveling to a waypoint in the main city area like Basilisk Gate or the main Baldur's Gate waypoint would work sometimes.
Ultimately it's just so disruptive to the gameplay and we spend more time restarting/reloading than actually playing that it just doesn't even feel worth it to play.
