Update: it turns out that this causes a way bigger bug. It turns out that when you reload a save the bug stops, however, you can now unequip the gloves and you still won't be silenced until he either dies or unequip and equip