I thought that the swamp taking over the farmlands was explicitly because of the effect of the Chaos Magic emanating from the crystal fragment.