That makes sense. I only used one charm (gold strength, dropped by the dragon rider Zandalor killed) and that was in the amulet (to equip the bastardsword from the knight by the castle a couple levels earlier than otherwise).