I have a solution that's worked for a friend and I. It's not a fix per se, but my friend and I are almost finished with Act 3 and we nearly have our golden dice!
The issue started once we entered the Lower City. As some of you have pointed out in this forum, it happens when entering the Lower City via door, portals, passageways etc., so any entrance that shows the 'Use mouse' icon over it (small mouse icon with the two gears below) to enter the Lower City will cause the crash.
Furthermore, using the Leave Camp button will also cause the game to crash if you used the Return to Camp button while inside the Lower City.
And finally, we have found the joinee's game crashes before they can even load into the host's server if their character's last location was in the Lower City.
Dealing with the last issue first: Firstly, the host should move the joinee's character out of the Lower City to Camp or a dungeon. Then, we found that joinee is always able to join when they drop their video quality settings to the lowest pre-sets from the options in the game's main menu. Then they can either join via Steam invite or lobby searching. This gets the joinee into the server. That's the first hurdle overcome.
The solution to the crashing is quite simple, when entering the Lower City, always enter via a map waypoint. Just open your map and click a waypoint.
Crashes still occasionally happen this way, but we've found that if the joinee also drops their video quality settings to their lowest before fast-travelling, this tends to avoid crashes quite consistently. The video quality settings can be returned to the joinee's preference after entering a dungeon (or any other entrance) that uses the 'Use' mouse icon. Just remember to lower the video settings again before fast-travelling back into the Lower City.
Once you enter the Lower City via a waypoint, you could increase your video settings again, but we found this increases the likelihood of a crash while exploring the area. I found playing at the lowest video quality was a small price to pay for the fun of playing through Act 3 with my friend.
Of course, accidents happen. It's very easy to forget about entering the Lower City via a waypoint. Your game will crash when this happens. But, all you need to do is Force Quit the game, get the host to move your character back to camp, drop your video quality settings back down the lowest from the Main Menu, and then rejoin the lobby (the same way I mentioned above).
I hope this helps. I hope it works for you as it has for my friend and I! Maybe when Patch 7 comes out in September this issue might be fixed, but until then, give this a try. Good luck!
(I should say that the joinee in our case was playing on the Mac version of the game with an M1 Pro 32GB and usually runs the game at ultra video quality where applicable with FSR turned off. We were also playing as origin characters, not Tav's, but since joinee Tavs remain the server even if the joinee themselves isn't in the server, I see no reason why this wouldn't also work for Tav playthroughs too).
