You can only go up. So if you start on the easiest setting, you could change it three times. Not sure of the official reason for this, but it does stop people going up to Hardcore when they level up so they can get the bonus skill points.