Charms were always meant to be a bonus, not a vital part of the adventure. There are so many charms available that you can afford to lose a few along the way.