I don't think it has something todo with leaving the fort. I had this before, while fighting in the arena but I already had more than ten hours on the clock and the saves where bigger than 7 mb.