Woah! Raze, that's crazy!
I had 10, maybe 12 charms in the whole game. I pick-pocketed one from the fighter's guildmaster and bought the rest.
I had a charm skill of 3, I think, it was enough for me.