Sounds more like a glitch than doing something wrong. Did you try teleporting to the homestead, to see if a map transition would help? Or dismiss and rehire Madora (though that would a longer workaround than quitting and restarting).