Update, something is able to remove permanent Clear-Minded during a fight as well. I have to reequip the item to get it back.