Same thing on the PC for me. Only ever started when I got to Rivington. Most of my current play-through was done after patch4 dropped, but I never saw anything like this in Acts I or II.
Walking around Rivington and Lower City can turn into a real test of patience because the game will hang for nearly a minute at times when you enter a new "sub-area" of the map. Also happens on fast traveling and after leaving camp after a long rest.
I've seen this when simply walking around inside the Circus, entering "named" buildings, entering Wyrm's Rock from Lower City, fast-traveling to the Sharess' Caress waypoint from Rivington, etc. It seems to be related to, well, entering a new section of map and probably to loading its assets because it feels like this gets much more pronounced the larger the "sub area" you're entering is.
And I'm not on a slow PC, by any means. i7-14700KF, 32 GB of fast DDR4 RAM, RTX 4070 and the game is installed on an M.2 SSD.
This really could use a fix - and not the DIY-fix that's been posted, because I don't want to have to mess around with config-files and/or the console.
It’s been 3 days and I haven’t heard so much as a “we are very sorry and are aware of the problem and working to fix it”.
According to the user who posted the DIY-fix, Larian are aware of this and are working on it. With so many folks owning the game, I would imagine the daily amount of bug-reports Larian receive is quite substantial and I'd rather have them work on the problems than sending a personalized "we're sorry" message to each and every one who reports a bug.

S.