Same in my case, the save files started to grow in size in Driftwood, had no issues at Fort whatsoever. It's not about the location but the saves themselves and some sort of memory leak. It gets only worse, to the point that it keeps trying to save the game while you can't quit or load another one.
