I agree, inventory cleanup does appear to decrease the freezing. It's still there and still annoying, but it makes it more manageable.

I haven't gone so far as to dump an entire inventory to camp, I may try that and compare.

I haven't noticed a difference with spellcasters. I'm generally switching between a bard and fighter and the freeze duration is similar.