I have noticed it only seems to happen to my characters when they camp while inside the abandoned village. If I go just outside the area or down the well or the cellar and camp, all seems fine.