That would be a good idea too. And maybe with a small chance to "ruin" the charm. Meaning charm slot lost or something.
So the player saves the game before having the charm removed, then reloads if this small chance happens to occur and it fails? Hmpf!
Whats the point of removable charms? The whole idea is that the player has to make a choice that can have both good and bad consequences; charm an item now for increased abilities at the risk of later finding a better charm/item with better base attributes.
If your able to remove a charm with a penalty, perhaps a charm could lose a grade/level in quality?