One thing I've noticed that might be related is that every save and everytime I open the map, it says "Fort Joy", even though I'm nowhere near there anymore (though I am still on the island).