If you're on PC it can be fixed with some* console commands; but it is quite annoying nonetheless.

* which I can't remember offhand, but I can probably dig them out.

J'aime le fromage.