Well, apparently this has been resolved by today's hotfix. It just randomly got better for me some point after Fort Joy. I also never had the freeze issue.