I'm encountering the same issues. As a guest player in my friend's game, we've racked up over 70 hours and are struggling through Act 3. I can't load into any area without setting my settings to the lowest possible and hoping for the best. We've even started splitting up, with me staying in the city and avoiding any teleports or "gear shaped" door teleports. If we need to enter a basement or similar area, my friend ventures alone and shares his screen via Discord so I can still participate. This situation is incredibly frustrating and causing me to miss a lot of the game. We had no issues in Act 1 or Act 2; we breezed through those acts.
I'm not sure if anyone will read this, but I wanted to share my thoughts. It might help if they split the city into smaller zones, effectively reducing the map size. We didn’t have problems until we reached the Lower City in Act 3. Crossing the bridge there caused a crash, and it keeps crashing every time, likely due to rendering too much at once. I’d prefer dealing with more loading screens than crashing continuously because of excessive rendering. This might not resolve the host/guest game issue entirely, but it could help.
