This needs more attention! I have been experiencing the same problem, exactly the same as OP, this seems to be a multiplayer Co-op issue that only affects the guest. I've tried pretty much all of the same methods too and nothing seems to fix it. I can also confirm with OP that this is still a problem even with patch 6, I only started and experience this at the tail end of Patch 5 Hotfix 17 and this was always only in Act 3 Lower City.
Larian please look into this, I love this game but we have 2 playthrough that have slowed to a crawl because of this. But for those that, if they can deal with constantly closing the game (P.S this works for me not sure if this works for you too.).
A temporary solution to get through Act 3 is to always send the guest player to camp before they join the game. For some reason I've been able to play through Act 3 by having my friend sending my character to camp, and when I join the game and leave camp, I can go through the lower city until I enter a new area and reenter the Lower City. You will need to do this every time you enter the lower city, and make sure that the guest Alt+F4.
