Once used, charms can not be removed, so try to stick to using silver or gold charms (the latter mostly available near the end of the game).