We're having exactly the same isssues. I don't have any problems on my side entering the lower city as host.
But in my partners game it is more or less gambling if her client loads correctly or fails with T-pose and missing UI elements / crashes or freezes in loadscreen at mostly every time 100%.
Restarting the games mosst times does not change anything. The only thing i had seen working sometimes ist leaving lower city, joining correctly again, and travel again.
But that is same gambling again, so that we are forced to stop right now in hope patch 1 can fix this maybe
