I've been dealing with this same issue on PS5. I first noticed it after the Gortash fight. I couldn't travel to lower city or undercity as everyone has so far described. I fortunately had a save before I entered the house of hope prior to fighting Gortash. With that save I can still freely travel to and from the lower city without any crashing. This should imply that the bug manifested somewhere in the House of Hope and/or the Gortash fight. Worth noting that this is after destroying the Watcher facility and fights at the Blushing Mermaid so presumably those are not at fault.

Since I'm now free of the crashing, I am going to first do the Gortash fight again since it is fairly short. What exactly should I be on the lookout for rpt this bug possibly manifesting in that fight and/or the House of Hope?