@All
The Cache is the problem (for most). Every scene you load, every character's wardrobe that is loaded, every weapon graphic that is loaded, those cached items never...ever clear. So it's a memory leak by design if you will. In Act 3 is really shines as a problem because of the quantity of NPCs vs the prior Acts. The Druid Grove's got nothing on Rivington for example.
The reason they don't clear is Larian's designers feel that when the characters pop up again, they will load faster since they're cached. Not taking into account the impact of Cache size on performance.