We're having exactly the same isssues. I don't have any problems on my side entering the lower city as host.
But in my partners game it is more or less gambling if her client loads correctly or fails with T-pose and missing UI elements / crashes or freezes in loadscreen at mostly every time 100%.
Restarting the games mosst times does not change anything. The only thing i had seen working sometimes ist leaving lower city, joining correctly again, and travel again.
But that is same gambling again, so that we are forced to stop right now in hope patch 1 can fix this maybe
Hey! You inspired me. Every time I've seen this bug talked about it's always the person NOT hosting that has an issue. I got the host to send me the files, and it works fine. I can zone in and out without any issues at all, now. My only guess is the non-host client can't download the assets from the host fast enough, and there's a window where it stops trying. Either way, give it a shot! I hope it fixes it for you, too.