I hold onto the charms because I hate wasting them in gear I won't use for the rest of the game.
As the final enemy sinks into oblivion and the ending sequence nears I think to myself "oh crap - wasted another set of 20 charms by never using them at all".
Unless you need the cash you might as well use them when you have them, because you will get that perfect piece of equipment that is worthy of them so close to the end that it won't make a difference - there is a lot less fighting in act 4 than in earlier acts.
